← All writing

How Do I Know an Agent Skill Is Actually Helping?

Three checks guide my Agent Skill updates: procedural adherence, final task outcome, and human review.

Once a Skill produces a tidy result, it is easy to call it useful. The agent followed a longer process, the transcript looks considered, and the final message says the work is done.

None of that tells me whether the Skill helped.

The useful comparison is the same realistic task with and without the Skill, or before and after a revision. Whenever I update one, I check the procedure the agent followed, the finished artifact, and my review notes. If I cannot point to a worthwhile difference, the extra instructions have not earned their place.

This article assumes the Skill already exists. For the underlying distinction between Skills, Tools, MCP, and Plugins, see The Agent Extension Stack. I have also written about how Skills changed my own work and where control lives in Matt Pocock’s Skills. Here I am concerned with the next problem: evaluation and maintenance.

Start with a failure, not a Skill

A weak evaluation begins with the Skill and asks whether the output looks good. A stronger one begins with something that went wrong in real work.

The failure needs to be specific enough to observe again. “The agent was inconsistent” is too broad. “The agent changed the article before reading the supplied sources,” “the visual repeated the nearby prose,” or “the review missed a required frontmatter field” gives me something I can test.

This matters because an agent may already handle much of the task well without help. The Agent Skills authoring guidance recommends asking whether the model would get an instruction wrong without the Skill. If the baseline already succeeds, adding more procedure can make the run longer without changing the result.

The history of this site’s article Skills offers a useful boundary. An earlier version of produce-article always sent a finished draft through the inline-visual phase. In a later revision, I changed that requirement: the workflow now asks whether a visual adds a relationship or evidence that the prose does not already carry. That revision records a change in instructions. It does not prove that articles became better or cheaper. (Yes, I use Skills to help turn my thoughts into readable articles, and every draft still has to survive me, the resident nitpicker.)

That history gives me an evaluation design, not a completed experiment. I could choose one article with a multi-stage relationship and one whose method is already clear as a numbered list, then run the old and current instructions against the same source packs. The comparison would show whether the decision gate removes redundant visuals without suppressing useful ones.

Run the same case twice

I start with two or three representative cases. One should be an ordinary task, one should expose the observed failure, and a third can test an awkward boundary in the procedure. The official evaluation guide recommends the same small starting point before building a larger suite.

Each case needs an expected artifact or outcome before the run begins. For an article workflow, that might include a source-backed draft, valid frontmatter, draft: true, media under the post’s directory, a release-tracker entry, and a successful build. The expected outcome should also name the judgment that a script cannot settle, such as whether the central claim is supported and the prose earns the reader’s time.

Then I run the same prompt and inputs in two clean contexts:

  1. a baseline without the Skill, or with the previous Skill version;
  2. a run with the Skill under evaluation.

The host, model, model version, available tools, permissions, and input files should stay the same. Otherwise I cannot tell whether the Skill caused the difference. For a small first pass, I keep the raw outputs and transcripts side by side. I do not need an evaluation platform to learn from three paired runs.

Three checks for every Skill update

These are the three checks I return to whenever I update a Skill. I keep them separate because a run can follow the instructions and pass a build while still producing work that needs substantial correction.

1. Procedural adherence

First I inspect what the agent did. Did it read the required sources before drafting? Did it preserve the slug? Did it perform the specified check once the artifact was stable? A transcript is useful here because it can reveal skipped steps, loops, and work that added no value.

Procedure is not the final goal, though. A run can follow every instruction and still produce the wrong result. This check tells me whether the Skill steered the work as intended. It does not tell me whether those instructions were good.

2. Final task outcome

I check the finished artifact against the expected outcome I defined before the run.

Mechanical checks should disagree with the agent when the artifact is wrong: the file exists, the schema accepts the frontmatter, links resolve, expected sections are present, and the build passes. For code, the equivalent may be a focused test or type check. I use scripts for these conditions when possible because a second language-model opinion is weaker than a deterministic check.

The remaining question is whether the artifact solves the task. A technically valid article can still make an unsupported claim. A passing code change can satisfy the test while making the interface harder to use. That part belongs in human review.

3. Human review

Mechanical assertions only cover the properties I remembered to encode. A human reviewer can catch a technically correct answer that misses the point, a source boundary that has been blurred, or a workflow that is unpleasant to use.

When practical, I compare the two artifacts without labels and review them against the same questions. Which one is more accurate? Which one needs fewer corrections? Which one makes its evidence easier to inspect? Blind review does not remove judgment, but it reduces the temptation to prefer the output produced by the more elaborate process.

The review should leave specific notes. “The Skill version kept the claim within the supplied evidence” can guide a revision. “This feels better” cannot.

Decide what the result earns

The outcome of an evaluation is not always “keep” or “delete.” I use four decisions:

Decision Evidence
Keep The Skill improves the task outcome and needs no more correction than the baseline.
Revise The Skill helps, but instructions are skipped, duplicated, or ambiguous.
Narrow The Skill helps only a particular type of task.
Remove The baseline matches or beats it on task completion and human review.

Removing a Skill is a valid result. So is keeping a short checklist while deleting the orchestration around it. The evaluation should preserve useful behavior even when that means deleting most of the Markdown.

The smallest repeatable loop

I use this loop whenever I update a Skill:

  1. Record one observed failure.
  2. Define the expected artifact or outcome.
  3. Run the same case with the previous and revised Skill, or without and with it.
  4. Inspect procedural adherence and run the task checks.
  5. Review the artifacts and record the corrections they need.
  6. Keep, revise, narrow, or remove the Skill.

Two or three cases are enough to begin. If the decision remains uncertain, I add the next case that can distinguish between the options. I expand the suite only after the small loop reveals a recurring failure worth checking.

I keep a Skill revision when the comparison shows a useful change in the work. If I cannot say which check improved, I keep the earlier version or revise the instructions again.