Claude Code Can Now Grade Your Own Skills (I Finally Have Proof Mine Work)

I run somewhere north of a dozen Claude Code skills across this business. One decides whether a lead counts as B2B and should never get consumer marketing. One drafts replies to inbound questions in my own voice. One audits calendar invites for a privacy setting I got burned by once already. Every one of them was "tested" the same way: I read the output a few times, it looked right, I shipped it.

That was the entire verification story for skills that touch real customers and real money. Until this week there was no way to ask a skill "did you actually help, or would Claude have gotten this right anyway?" Anthropic just shipped an answer to that question directly into the CLI.

What shipped

claude plugin eval landed in Claude Code v2.1.269 on September 11, 2026. It runs your plugin or skill against a suite of test cases you write (or have Claude write for you with claude plugin eval init), scores each one with a grader, and, this is the part that matters, runs the same cases again with your plugin not loaded so you can see what it actually contributed.

That second run is the whole point. A skill can pass every test you throw at it and still be worthless, because Claude might have produced the same answer without any help. The docs put it more bluntly than I would have: "if a case scores 1.0 both with and without the plugin, the plugin isn't what made it pass."

That example is straight out of Anthropic's own quickstart, and it's a good illustration of the failure mode this catches: a plugin that scores perfectly on its own tests but only moves the needle a third of the time versus a bare Claude session. Without the baseline run, you'd never know the difference between "this skill works" and "Claude was already going to do this."

How a suite is actually built

An eval suite lives in an evals/ folder next to your plugin. Each case is a directory with a prompt and one or more graders. There are six grader types, and the split between them matters for cost: four are free because they inspect the transcript or filesystem directly, and two call a judge model, which is where the bill comes from.

A minimal case is two files. prompt.md holds the request a real user would type, plus frontmatter for turn and time limits:

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---
 
Write me a commit message for this change: I renamed getUser to
fetchUser and updated the three call sites.

And graders/skill-fired.md checks that your skill is the thing that actually produced the answer, not incidental Claude behavior:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"'
---

Run claude plugin eval . from the plugin root and you get a summary table with WITH, W/OUT, and Δ per case, plus a self-contained HTML report you can hand to someone who's never seen the suite.

The part that actually changes my workflow

The CI gate is the reason I care about this beyond curiosity. You can pin a threshold and a model, run headless, and fail the build the moment a skill regresses:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

That's a real gap this closes. Every skill in this repo has a comment somewhere saying it was "proven to fail before being trusted to pass," which up to now meant me hand-injecting a bad input and eyeballing whether the guardrail caught it. Pinning the model matters too: a skill that quietly stops working the day a new default model ships is exactly the kind of regression nobody notices until a customer does.

The first place I'm pointing this is the B2B-exclusion check that decides whether a lead gets consumer nurture emails, because getting that wrong once already cost a real conversation with a real prospect. A tool_used grader that confirms the check actually ran, paired with an llm grader on the outcome, against the no-plugin baseline, tells me something a manual read-through never could: whether the skill is doing the work, or whether Claude was going to get it right regardless and I've been trusting a placebo.

If you're running any nontrivial number of Claude Code skills or plugins, that's the question worth asking about every one of them. This is the first time there's been a built-in way to actually answer it.

Sources: Test plugins with evals · Create plugins · Anthropic Adds Plugin Evals to Claude Code (MarkTechPost)