Anthropic Adds First-Party Plugin Evals to Claude Code: Six Graders and a CI Gate

Anthropic shipped a native plugin evaluation system in Claude Code v2.1.269 on September 11, giving plugin developers the first first-party tool to measure whether a skill actually triggers — and whether it makes any measurable difference when it does. The feature, invoked as claude plugin eval, runs each plugin against a suite of test prompts, scores what Claude produces, and compares the result with a parallel run where the plugin is not loaded at all. The difference between those two scores, called Δ (Delta), is the only number that proves a plugin contributed to the outcome rather than rode along as a passenger.

The release addresses a long-standing gap in the Claude Code plugin ecosystem. Until now, developers shipping skills — the natural-language-triggered components that steer Claude toward specific workflows — had no first-party mechanism to verify that a skill actually fires on the prompts their users would naturally type, or that it improves the result over what bare Claude would do anyway. The existing claude plugin validate command checks manifest syntax and schema. It cannot check behavior.

Read more: Claude Code's SendFeedback Tool Lets AI Draft Its Own Session Failure Reports

How the Scoring Works: Arms, Runs, and the Delta That Matters

The evaluation system is structured around what Anthropic calls a two-arm design. Every test case runs twice: once in the with-arm, where the plugin is loaded, and once in the without-arm, where no plugin is present. Each arm runs the same prompt three times by default to reduce the noise inherent in non-deterministic agent behavior. A single run of an agentic system tells you little; three runs yield a mean score with enough stability to trust.

Each case's score in a given arm is the fraction of graders that passed, optionally weighted, averaged across the three runs. The Delta is the with-arm score minus the without-arm score. A Delta of +0.67 — the example in Anthropic's official documentation, representing a case that scored 1.00 with the plugin and 0.33 without — means the plugin raised performance by 67 percentage points on that prompt type. A Delta near zero means the plugin is not contributing: Claude would have produced an equivalent result without it.

This design matters more than it might appear. AI developer teams have historically evaluated plugin quality by checking whether Claude's output looked correct, a bar that does not distinguish plugin-driven correctness from baseline model capability. The Delta score imposes a stricter standard: a plugin must be additive, not merely present. A case that scores 1.00 in both arms is, by Anthropic's explicit framing, a case the plugin is not responsible for passing. According to the documentation, the most common first finding when developers run evals is a Delta near zero paired with a failing tool_used: Skill grader — meaning Claude is not choosing the skill on natural phrasing at all.

Six Grader Types: Four Free, Two Billed to Your Account

The evaluation system provides six grader types, split cleanly by cost. Four graders — regex, tool_used, tool_order, and file_exists — are computed from the session transcript and files on disk and cost nothing beyond the agent run itself. The other two — llm and baseline — call a judge model and add to the API bill.

regex applies a JavaScript regular expression to the session output, the full transcript, or a specific file Claude created during the run. tool_used checks whether a particular tool was called a specified number of times, with optional regex matching against the call's input. tool_order verifies that one tool was called before another — relevant for workflows where sequence matters. file_exists confirms whether a file matching a given glob was created by Claude during the session (not merely modified or present from setup).

llm graders ask a judge model to vote on whether the session output satisfies a prose rubric written as concrete PASS and FAIL conditions. Anthropic's implementation uses majority voting across three judge calls to reduce individual call variance — a reasonable mitigation given that LLM judge consistency degrades for longer outputs and loosely specified rubrics. The baseline grader takes a reference transcript and asks the judge whether the current run satisfied the criteria at least as well as that reference, providing a comparison anchor rather than an absolute standard.

Graders are plain Markdown files whose frontmatter sets the type and options. An arm field can restrict a grader to the with-arm only — this is how the system handles tool_used: Skill checks, which cannot meaningfully pass in the without-arm where no skill exists to fire. Anthropic excludes arm-restricted graders from the comparative score in both arms and reports them as indicators only, preventing them from artificially deflating the without-arm score and distorting the Delta.

CI Gate: Threshold, Cost Ceiling, and Exit Codes

For teams that want to block deploys on quality, the system provides a documented CI invocation:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

The --threshold parameter sets the minimum with-arm score required for a case to pass; any case below it causes the command to exit with code 1, blocking the build. The --max-cost-usd flag sets a ceiling on estimated list-price cost — though Anthropic notes explicitly that runs already in flight when the ceiling is hit will complete, potentially exceeding the ceiling by those runs' worth. When the ceiling stops the suite early, the command exits with code 2 and writes partial results to the JSON output file, allowing CI scripts to distinguish a quality failure (exit 1) from a budget failure (exit 2).

Pinning both --model and --judge-model is important for CI reproducibility. Without pinning, a model rollout during an active CI pipeline could produce score changes that look like plugin regressions but are actually model behavior shifts.

The cost model scales with suite complexity. A single case running at the default three runs per arm produces six total agent runs plus three short judge calls per llm or baseline grader per run. Anthropic's documentation includes a worked example — one case, six runs, two graders — producing an estimated cost of $0.41 and taking 74 seconds. These figures are company-provided estimates and will vary with model selection, context length, and the number of judge-graded assessments.

What claude plugin validate Could Not See

The distinction between claude plugin eval and the pre-existing claude plugin validate command illuminates what was missing before this release. claude plugin validate performs static analysis: it checks whether a plugin's manifest conforms to schema, whether required fields are present, and whether file references are valid. This catches authoring errors and structural mistakes before a plugin is installed. It cannot run the plugin against a model or observe what the model actually does.

The gap this creates is consequential. A skill's most important attribute is its description field, which is what Claude reads when deciding whether to invoke the skill for a given user prompt. A description that is syntactically valid but poorly matched to natural user language will pass validate cleanly and then fail to trigger in practice — the plugin ships, developers assume it works, and the failure mode is invisible until a user notices the skill never fires. claude plugin eval catches this before deployment by running realistic prompts and checking whether the skill is actually invoked.

Smart Reports and the Enterprise Governance Layer

Separately from the eval system, Anthropic has launched Smart Reports in beta for Enterprise plan customers. Smart Reports analyze team-level Claude Code usage — what work is getting done, what it costs, where sessions run into friction, and which repeated patterns are worth packaging as shared skills. The launch positions Anthropic's enterprise offering to go beyond per-developer tooling into organizational observability: administrators can measure adoption, identify high-friction workflows, and use friction patterns as signals for where new skills or plugins should be built.

The combination of plugin evals and Smart Reports represents a coherent governance layer for enterprise Claude Code deployments. Evals provide pre-deployment quality assurance at the plugin level; Smart Reports provide post-deployment usage and cost visibility at the team level. Together, they give enterprise buyers the instrumentation to treat Claude Code as managed infrastructure rather than a developer convenience — a shift that matters for procurement, compliance, and IT governance.

Read more: Claude Code Desktop Gains /resume, Ending the Context-Loss Tax for Developers

Competitive Context: A Gap the Community Had Already Noticed

Third-party developers had been building partial solutions to this problem for months before this release. Libraries including bkper/claude-eval, sjnims/cc-plugin-eval, and the coder-eval Python package on PyPI each addressed aspects of the behavioral testing problem — checking whether skills trigger, scoring outputs against LLM judges, gating CI on quality metrics. The existence of multiple independent implementations of the same pattern confirms that the gap was real and that demand existed across the developer community.

None of those tools are first-party, and none include the with/without ablation design that makes Delta meaningful. A third-party eval framework can check whether a plugin produces correct output; it cannot, without additional engineering, distinguish plugin-caused correctness from zero-shot correctness. The Delta score is the methodological addition that first-party integration enables.

Competing AI coding platforms have not yet shipped equivalent systems. Neither GitHub Copilot's extension ecosystem nor Cursor's tooling includes a documented first-party behavioral evaluation framework for extensions, based on current public documentation. Microsoft's Copilot Studio includes a server-side Evaluation API for agent quality testing in Azure DevOps pipelines, but the architecture differs substantially — it is server-evaluated against draft agents rather than locally executed against a behavioral baseline — and it addresses enterprise agent deployment rather than developer-authored CLI skills.

Limitations and What Developers Should Check

Several characteristics of the system warrant attention before integrating it into CI pipelines.

LLM grader scores are not fully stable across runs. Anthropic's documentation notes that llm grader verdicts can differ between runs, and the variance increases with output length and loosely specified rubrics. The three-vote majority system reduces but does not eliminate this variance. Teams should treat llm grader pass rates as probabilistic estimates rather than deterministic quality signals, and should calibrate rubrics with concrete PASS and FAIL conditions rather than open-ended criteria.

Rate limits can produce spurious failures. Anthropic explicitly warns that if an account reaches its plan usage limit or an API rate limit mid-suite, subsequent runs will fail, score zero, and produce results that resemble a plugin regression. The NOTES column in the output and the cases[].arms.with[].error field in the JSON output distinguish limit errors from genuine failures — teams should check these fields before acting on a failed CI run.

The --max-cost-usd ceiling is an estimate ceiling, not a hard cap. Runs in flight when the ceiling is reached will complete. Teams building cost-controlled CI pipelines should set the ceiling below the actual budget limit to create headroom for in-flight runs.

The system requires ANTHROPIC_API_KEY credentials in the CI environment. Every eval run and every judge-graded call is billed to the associated plan or API account. For large suites, costs can accumulate quickly; the --ablation none flag halves cost by skipping the without-arm for runs where Delta is not needed.

On Windows, Bash-granting eval suites are unsupported without WSL2 because the OS-level sandbox that confines shell commands (bubblewrap on Linux) has no native Windows backend.

What the Delta Score Changes About Plugin Development

The Delta score — the difference between a plugin's performance and what Claude would do without it — is not a new concept in machine learning evaluation. Ablation studies that remove a component to measure its contribution are standard research methodology. What is new is the application of that methodology as a first-party, developer-facing quality gate for AI platform extensions, integrated directly into a CI pipeline with cost controls and exit codes.

The practical consequence is a change in what it means to ship a Claude Code plugin. Before v2.1.269, shipping a plugin meant confirming that its manifest was valid and that the skill triggered on a few test prompts in interactive sessions. After v2.1.269, it means producing a Delta score — a measurement of how much the plugin improves Claude's behavior relative to no plugin at all — and gating deployment on that score meeting a defined threshold.

That standard is stricter, and it should be. The Claude Code plugin ecosystem now includes hundreds of community-authored extensions covering categories from PR review to financial analysis to data analytics. As that ecosystem grows, the ability to verify that plugins deliver measurable value — rather than just appearing to work — becomes important for developers choosing which plugins to adopt, for organizations deciding which plugins to approve for enterprise deployment, and for Anthropic maintaining the quality of a catalog that carries its name. The graders and the CI gate are the mechanism. The Delta score is the claim: the plugin made a difference that can be measured.