LLM evals for product features: from hand-read traces to a CI gate

LLM evals are automated tests for features built on language models: a fixed set of inputs, a grader that decides pass or fail for each output, and a score you track across every prompt, model and code change. They replace the "looks good to me" check that works for the first demo and stops working once a feature has real users and a second developer.

This article describes a path that holds up in practice: read real traces by hand, turn what you find into a failure taxonomy, pick the cheapest grader that can detect each failure, validate any LLM judge against human labels, measure reliability with pass^k as well as pass@k, and run the result as a regression suite in CI.

Why do LLM features need evals?

Because model outputs vary between runs and every prompt tweak can fix one case while breaking three others you are not looking at. Without a fixed test set you only find out from users.

Unit tests assume a function returns the same output for the same input. An LLM feature doesn't, and its failures are rarely crashes: a summary that drops the one sentence that mattered, a support answer that invents a refund policy, a classifier that drifts after a model upgrade. Evals turn those into numbers you can compare. Anthropic's Demystifying evals for AI agents (January 2026) makes the start sound small on purpose: "20-50 simple tasks drawn from real failures is a great start". You don't need a benchmark; you need the cases that already went wrong.

How do you start? Error analysis on real traces

Start by reading traces, not by choosing metrics. Review around 100 real interactions, write a free-text note on each problem, then group the notes into a small taxonomy of failure modes and count them.

Hamel Husain's evals FAQ (updated September 2026) describes the method in qualitative-research terms. Open coding: annotators write free-text notes about what went wrong in each trace. Axial coding: the notes are grouped into a failure taxonomy and counted. He suggests starting with about 100 diverse traces and reviewing "at least 30 traces yourself before asking the agent to suggest failures", and he expects "60-80% of development time on error analysis and evaluation". The counts matter: they tell you which failure is worth a grader first.

Two organizational points from the same FAQ. Appoint one "benevolent dictator", a domain expert who makes the final call on what counts as good, so labels stay consistent. And expect your criteria to move: Shankar et al. call this criteria drift, the observation that "users need criteria to grade outputs, but grading outputs helps users define criteria". Re-read traces after the taxonomy exists; it will change.

Which grader should you use: code, model or human?

Use the cheapest grader that reliably detects the failure: code for anything checkable by rule, an LLM judge for judgment calls, and humans to calibrate the judge and to review what the automated graders can't decide.

The strengths and weaknesses in the table are the ones Anthropic lists in its evals article. One more rule from the same source prevents a whole class of flaky tests: grade "what the agent produced, not the path it took". An agent that reaches the right result by a different sequence of tool calls should pass. When the output is a fixed choice (a category, a priority, a route), you can avoid free text altogether and grade an exact match; that pattern is described in typed decisions instead of prose.

How do you build an LLM-as-judge you can trust?

Give each judge one failure mode and a binary pass/fail verdict, then measure it against human labels with true positive and true negative rates before you trust its numbers.

Binary verdicts, in Husain's words, "force clearer thinking and more consistent labeling" than 1–5 scales, where adjacent points mean different things to different people. One judge per failure mode keeps the prompt short and the verdict debuggable:

You are grading one failure mode: UNSUPPORTED_POLICY_CLAIM.

FAIL if the answer states a refund, shipping or warranty rule
that does not appear in the provided policy excerpts.
PASS otherwise, including when the answer says the policy
does not cover the question.

Policy excerpts:
{excerpts}

Answer:
{answer}

Reply with JSON: {"reason": "one sentence", "verdict": "PASS" | "FAIL"}

Then validate it. Husain recommends labelling 100 to 200 examples per failure mode and splitting them: 10–20% as few-shot examples in the judge prompt, 40–45% as a development set to refine the prompt, and 40–45% as a held-out test set. Measure the true positive rate (of the outputs humans marked as failures, how many the judge also failed) and the true negative rate (of the passes, how many the judge also passed). Raw agreement hides a judge that passes everything when failures are rare.

Known biases are worth testing for. The MT-Bench study (Zheng et al., 2023) found strong LLM judges reached "over 80% agreement" with human preferences, but also documented position, verbosity and self-enhancement biases. If the judge compares two outputs, swap their order and check the verdict holds.

What is the difference between pass@k and pass^k?

pass@k is the chance that at least one of k attempts succeeds; pass^k is the chance that all k succeed. pass@k fits tools where a user can retry or pick the best answer; pass^k fits features that must work every time.

Anthropic defines pass@k as "the likelihood that an agent gets at least one correct solution in k attempts" and pass^k as "the probability that all k trials succeed". pass^k was introduced by τ-bench (Yao et al., 2024), which found that even strong function-calling agents were inconsistent: gpt-4o succeeded on fewer than half the tasks, with pass^8 below 25% in the retail domain. The two metrics diverge fast. As an illustration, assuming independent trials with a 90% success rate each: pass@5 is 1 − 0.1⁵ ≈ 99.999%, while pass^5 is 0.9⁵ ≈ 59%. A support bot that answers the same question correctly nine times in ten will get it wrong for a noticeable share of users. Run each eval case several times and report both.

How do you run evals in CI?

Split the suite into a regression set that must stay near 100% and a capability set that is allowed to fail, run the regression set on every change, and gate merges on it.

• Regression evals hold the cases that already work. Anthropic's guidance is that they should have "a nearly 100% pass rate"; a drop is a bug.
• Capability evals hold the cases you are trying to make work. They "should start at a low pass rate" and graduate into the regression set once they pass reliably.
• Every production failure becomes a case. The trace goes into the set with its expected behavior before the fix is written, in the same spirit as a regression test that must fail when the fix is reverted, described in harness engineering for coding agents.
• Pin everything that isn't under test: model id, temperature where the API allows it, judge prompt version, and for agent evals the execution environment.

That last point is not pedantry. Anthropic's Quantifying infrastructure noise in agentic coding evals (February 2026) found that on Terminal-Bench 2.0 the difference between the most and least resourced configurations was 6 percentage points, with the same model and harness. If your CI runners change size, your agent's pass rate can move without any code change. Compare runs on identical infrastructure, and treat small differences with suspicion.

When evals mislead

Evals mislead when the metric is generic, the judge is unvalidated, the set is stale, or the noise is larger than the difference you are reading.

• Generic metrics. Off-the-shelf "helpfulness" or "coherence" scores rarely map to your failure modes. Husain: "Generic evaluations waste time and create false confidence when you use them as quality measures."
• An unvalidated judge. A judge that agrees with itself tells you nothing. Without TPR and TNR on held-out labels, the score is the judge's opinion.
• Saturation. A capability set at 100% no longer measures progress. Move its cases into regression and write harder ones from new traces.
• Too few trials. With a few dozen cases and one run each, a swing of a few points can be noise. Repeat runs and look at the individual cases that flipped.
• Skipping the transcripts. Anthropic's advice is blunt: "You won't know if your graders are working well unless you read the transcripts."

LLM evals checklist

1. Read about 100 real traces, at least 30 yourself, and write a free-text note on every problem.
2. Group the notes into failure modes and count them; build graders for the most frequent first.
3. Name one quality owner who makes the final call on labels.
4. Use code graders wherever a rule works; reserve LLM judges for judgment calls.
5. One binary judge per failure mode, validated with TPR and TNR on a held-out labelled set.
6. Run each case several times and report pass^k for features that must work every time.
7. Keep a regression set near 100% and gate merges on it; let the capability set fail.
8. Turn every production failure into a case before writing the fix.
9. Pin model, prompts and infrastructure so a score change means a behavior change.

If you're adding evals to an LLM feature and want help setting up the first suite, see AI engineering.

Sources

1. Anthropic: Demystifying evals for AI agents (2026)
2. Hamel Husain: LLM evals FAQ (updated September 2026)
3. Shankar et al., Who Validates the Validators? (2024)
4. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
5. Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction (2024)
6. Anthropic: Quantifying infrastructure noise in agentic coding evals (2026)