Claude Code Can Build Evaluations and Improve Apps Against Held-Out Tests

Anthropic's claude-api skill adds two evaluation workflows for Claude Code: /claude-api build-eval creates an evaluation inside a codebase, and /claude-api hillclimb tries to improve an application against that evaluation one change at a time. Anthropic says the hillclimb workflow uses a held-out set to catch overfitting (Anthropic).

The guide is for developers who already have an application and want to measure a concrete change, such as improving performance or lowering cost while holding performance steady. It does not make a benchmark automatically representative of production. The user still chooses the task distribution, reviews the cases and grader, and decides which code or configuration Claude may change (Anthropic). To test a Claude Code plugin or skill rather than an application, see Claude Code's plugin evals.

Key Takeaways

• Two commands: /claude-api build-eval builds an evaluation inside your codebase, and /claude-api hillclimb improves your application against it one change at a time.
• Update first: the claude-api skill ships inside Claude Code, so running claude update gets you the latest version of both commands (Anthropic).
• You approve the evaluation: build-eval interviews you and pauses for approval at specific points, including the examples and the grader.
• Held-out test set: if the train set improves but the test set is flat, Claude suspects overfitting and reverts the patch.
• Noise check: if the gain is within noise, Claude says so and recommends against merging.
• Anthropic's example: on 14 held-out tickets from a 44-ticket customer support benchmark, the final configuration scored 90.5% against the original setup's 78.6%, at about one fifth of the cost (Anthropic).

Build an Evaluation From Real Work

Running /claude-api build-eval interviews the developer, then helps assemble an evaluation set, grader, runner, and baseline. The skill can draw cases from production transcripts, bug reports, support tickets, examples supplied by the developer, or synthetic cases based on the codebase. It shows the inputs for review and pauses at approval points before running the baseline (Anthropic).

The article's design principle is to mirror important production tasks rather than select cases simply because the current model fails them. Anthropic advises using tasks a domain expert can judge consistently, with the criteria stated clearly enough for the grader to check. The baseline also needs headroom: when the baseline already scores about 95% or higher, the skill warns the developer and steers the hillclimb toward cost or latency rather than quality (Anthropic).

The skill checks whether the grader gives consistent results on repeated runs and watches for infrastructure problems such as timeouts, API errors, or truncated answers. It reports a baseline score with a confidence interval and recommends reading scored transcripts before trusting the evaluator (Anthropic).

Improve One Change at a Time

Running /claude-api hillclimb asks what to optimize and which changes it may make. The allowed surface can include a system prompt, skills or instruction files, tool descriptions, model choice, effort level, API parameters, or harness code. Claude proposes one patch per round and compares the changed application against the evaluation (Anthropic).

To reduce overfitting, the workflow randomly separates the evaluation into training and test cases. Claude can inspect the training cases, but the held-out test set is kept out of its reach. It reverts a patch when training improves but the test score stays flat, and keeps a patch when both sets improve. If the measured gain is within the evaluation's noise, the guide says not to merge it (Anthropic).

What to Check Before Trusting the Result

An evaluation can still diverge from real production traffic. Anthropic warns that a system can overfit to selected examples or exploit details in the harness instead of improving the application for real users. The guide recommends:

• Use cases that matter in production, including concrete failures from traffic, tickets, and bug reports (Anthropic).
• Keep a test set that the optimization loop never sees during tuning (Anthropic).
• The hillclimber may read failing transcripts, but it must never paste the failure content into the prompt it is tuning (Anthropic).
• Make it structurally difficult for the system being evaluated to access the test answers (Anthropic).
• Treat a change as meaningful only when its test-set gain exceeds evaluation noise (Anthropic).

These safeguards make the workflow a way to run controlled experiments in an application codebase, not a guarantee that a model change will generalize. Developers should inspect the cases, grader, and transcripts before acting on a reported gain (Anthropic).

Corrections and Updates

• September 30, 2026: This page said the headroom threshold applied to "a top model" and that failing transcripts must not be pasted "into prompts used by the hillclimber". Anthropic's post ties the threshold to the baseline run of the developer's own configuration, and says the hillclimber may read failing transcripts but must never paste the failure content into the prompt it is tuning. Both sentences are corrected above.

Sources

• Anthropic, “Automating eval design and hillclimbing with Claude,” September 28, 2026: https://claude.dev/blog/automating-eval-design-and-hillclimbing/