Claude Code Adds Plugin Evals for Regression Testing and No-Plugin Baselines

Claude Code now has a first-party testing workflow for plugins and skills: claude plugin eval. The new command lets authors create test cases, run a plugin against realistic prompts, score the results, and compare the outcome with a no-plugin baseline. (ClaudeDevs on X; Claude Code plugin evals documentation)

The feature targets a practical problem with extensible agents: a plugin can appear useful while Claude would have produced the same result without it, or a model update can quietly change whether a skill fires. Evals make that contribution measurable and give teams a repeatable way to catch regressions before they ship. (Claude Code plugin evals documentation)

Key Takeaways

• claude plugin eval init can draft a suite of prompts and graders from a plugin.
• claude plugin eval . runs the suite in fresh, isolated non-interactive sessions.
• A no-plugin arm is included by default, so teams can measure what the plugin actually contributes.
• Reports show scores, grader verdicts, runs, and estimated cost, with an HTML report written locally and optionally published.
• Plugin authors can gate changes in CI, according to Anthropic's documentation.
• The feature requires Claude Code v2.1.269 or later, and model calls count against plan usage or an API bill.

What an Eval Suite Measures

An eval case is a realistic prompt that a user might send to Claude when the plugin should help. Each case can have one or more graders: a pass/fail check over the reply, a tool call, a created file, a regular expression, or a rubric judged by another model. (Claude Code plugin evals documentation)

The workflow is designed to test behavior rather than only package validity. claude plugin validate checks plugin files for syntax and schema errors; claude plugin eval asks whether the installed plugin reliably steers Claude toward the desired outcome. (Claude Code plugin evals documentation)

Start with plugin eval init

From a plugin directory, an author can run:

claude plugin eval init

Claude Code reads the plugin and asks what a good result looks like. It proposes prompts that should and should not trigger the plugin, designs graders, pilots the suite, and writes cases under the plugin's evals/ directory. Authors can also create a bare case and write its prompt and graders manually. (Claude Code plugin evals documentation)

The documented requirements are:

• Claude Code v2.1.269 or later.
• A plugin directory with plugin.json or .claude-plugin/plugin.json, or a skills-directory plugin.
• The same authentication and model provider used by normal Claude Code sessions.

Because initialization asks Claude to inspect the plugin and propose tests, init is itself a model call. Anthropic says eval runs, judge-scored graders, and initialization count against the plan's usage limits or the user's API bill. (Claude Code plugin evals documentation)

Run with and without the Plugin

Running the suite from the plugin root is straightforward:

claude plugin eval .

For each case, Claude Code starts a fresh isolated non-interactive session with the plugin loaded, sends the prompt, and lets the agent work until it finishes or reaches the case's turn or time limit. Graders then inspect the final reply, transcript, or files the agent created. (Claude Code plugin evals documentation)

The default comparison is the important part. Each case is also run with no plugin loaded, producing a WITH score and a W/OUT score. Their difference, Delta, is the plugin's measured contribution. A high score with the plugin is less informative when Claude achieves the same result without it. (Claude Code plugin evals documentation)

One case runs three times by default because a single agent run is noisy. The no-plugin arm means the full default suite can use six agent runs for one case, before any judge calls. Authors can use --runs 1 --ablation none for a cheaper, single-arm iteration, then return to the default three-run comparison before trusting a change. (Claude Code plugin evals documentation)

Read the Report

After a run, Claude Code prints a summary with the case score, plugin and baseline scores, difference, number of runs, estimated cost, and notes about failing graders. It writes a detailed HTML report under evals/results/; accounts that support report publishing can also receive a hosted report, while --no-publish keeps the report local. (Claude Code plugin evals documentation)

The report is useful for more than a pass percentage. It can show which grader failed, the model judge's explanation, the transcript excerpt it evaluated, and whether the plugin's skill was invoked. That makes the next edit more concrete: adjust a skill description, prompt, case, or grader, then rerun the comparison.

CI and Trust Boundaries

Anthropic documents plugin evals as a way for teams to gate plugin changes in CI. The case files are plain files under evals/, and teams can keep result artifacts out of version control while running the suite as part of a change check. (Claude Code plugin evals documentation)

The command is not risk-free. Eval sessions run with the plugin loaded, and the documentation warns that plugin hooks and MCP servers run as the user. Authors should evaluate only plugins they trust. Model-based graders also consume usage and can vary between runs, so a small pilot should not be treated as a definitive quality measurement. (ClaudeDevs on X; Claude Code plugin evals documentation)

What This Changes for Plugin Authors

Before plugin evals, teams could validate the package and manually inspect a few sessions, but that did not answer two operational questions: does the plugin improve the result, and does it keep improving after a model or plugin change? The WITH/W/OUT comparison gives authors a direct baseline, while case-level graders preserve the details hidden by one aggregate score. (Claude Code plugin evals documentation)

The most useful first suite is likely small: a handful of prompts that should trigger the plugin, a few prompts that should not, and graders for both the result and the expected tool or skill behavior. Expand it when a real regression appears rather than treating a large synthetic benchmark as proof of production reliability.

Bottom Line

claude plugin eval turns Claude Code plugin quality from an informal demo into a repeatable test loop. Authors can generate cases, run isolated sessions, compare the plugin against a no-plugin baseline, inspect an HTML report, and place the suite in CI. The trade-off is explicit: each run uses model calls, and teams must treat hooks and MCP servers as trusted code. (Claude Code plugin evals documentation)

Sources

• ClaudeDevs on X, “New in Claude Code: claude plugin eval,” September 11, 2026: https://x.com/ClaudeDevs/status/2098500999656923145
• ClaudeDevs on X, “Evals call the model...,” September 11, 2026: https://x.com/ClaudeDevs/status/2098501003666702344
• Anthropic, Claude Code Docs, “Test plugins with evals”: https://code.claude.com/docs/en/plugin-evals