Evaluating AI agents: grading trajectories, outcomes and reliability

An agent that says "your refund has been processed" and an agent that actually processed the refund look identical in a chat log. Evaluating agents means looking past the final message: at the outcome (what changed in the world) and at the trajectory (which tools were called, in which order, with which arguments). Grading only the final text is how agents pass evals and fail customers.

This article is the agent-specific follow-up to LLM evals for product features. It covers the vocabulary, the three kinds of graders, trajectory matching with agentevals, DeepEval and Ragas, why pass^k matters more than pass@1 for customer-facing agents, and how to keep the eval environment from lying to you.

Outcome and trajectory: the two things you grade

Anthropic's January 2026 guide gives a vocabulary worth adopting. A task is one test case with inputs and success criteria; each attempt is a trial. A grader scores some aspect of a trial. The transcript (also called trace or trajectory) is the complete record: outputs, tool calls, reasoning, intermediate results. The outcome is the final state of the environment. Their example: a flight-booking agent may say "Your flight has been booked", but the outcome is whether a reservation exists in the database.

τ-bench, one of the reference agent benchmarks, grades exactly this way: it compares the database state at the end of a conversation with an annotated goal state. That is the strongest single check you can build, because it does not care what the agent said, only what it did.

Code, model and human graders

Each grader looks at either the transcript or the outcome. Anthropic's guide splits them into three families, and a good agent eval mixes all three:

Scores can be combined as a weighted sum against a threshold, as a binary all-must-pass, or as a hybrid. My default for business agents: binary on safety-relevant code checks (no refund without verification, ever), weighted on everything else.

Trajectory matching: strict, unordered, subset, superset

When you have a reference trajectory, the question is how exactly the agent has to follow it. LangChain's open-source agentevals package names the four useful answers as trajectory_match_mode values:

Arguments are matched separately. By default agentevals requires identical arguments; tool_args_match_mode can be set to ignore, subset or superset, and tool_args_match_overrides takes per-tool rules, including a custom comparator (case-insensitive city names, for example):

from agentevals.trajectory.match import create_trajectory_match_evaluator

# Refund flow: the agent may look things up freely, but must make these calls.
evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="superset",
    tool_args_match_overrides={
        "issue_refund": ["order_id"],          # only order_id must match exactly
        "lookup_order": "ignore",              # any lookup arguments are fine
    },
)
result = evaluator(outputs=agent_messages, reference_outputs=reference_messages)
# {'key': 'trajectory_superset_match', 'score': True, 'comment': None}

Other frameworks cover the same ground with different names. DeepEval's ToolCorrectnessMetric compares expected and called tools, by name by default, optionally also by input parameters and output, with flags for ordering and exact match. Ragas offers ToolCallAccuracy (strict order by default, with a flexible mode), ToolCallF1 for precision and recall over calls, and AgentGoalAccuracy for when only the outcome matters. Pick one framework per project; the concepts transfer. My reviews of DeepEval and Ragas cover the trade-offs.

Grade outcomes first, paths only for invariants

The temptation with trajectories is to demand the exact path you would have taken. Anthropic warns against it: agents regularly find valid approaches the eval designer did not anticipate, so "it's often better to grade what the agent produced, not the path it took." Their own example is striking. Opus 4.5 solved a τ²-bench flight-booking problem by finding a loophole in the policy; it "failed" the eval as written while producing a better result for the user.

So I use trajectory checks for invariants, the few rules that must hold on every path, and outcome checks for success. Invariants are usually short and easy to express in code:

def check_invariants(calls: list[dict]) -> list[str]:
    """calls: tool calls from the transcript, in order: {"name": ..., "args": {...}}"""
    errors, verified = [], False
    for c in calls:
        if c["name"] == "verify_customer":
            verified = True
        if c["name"] == "issue_refund":
            if not verified:
                errors.append("refund before customer verification")
            if c["args"]["amount"] > 500:
                errors.append("refund above approval limit")
        if c["name"] in {"delete_account", "send_bulk_email"}:
            errors.append(f"forbidden tool: {c['name']}")
    if len(calls) > 25:
        errors.append(f"{len(calls)} tool calls, budget is 25")
    return errors

This is binary and cheap, it runs on every trial, and it does not punish creative paths. Combine it with the outcome check against the database, and you have caught the two failures that matter most: the agent did the wrong thing, or it did the right thing in a forbidden way. For deciding which actions need a human gate in the first place, see human in the loop for AI agents.

pass@k and pass^k: one trial is not a measurement

Agents are non-deterministic, so a task that passes once may fail on the next run. Two metrics describe this. pass@k is the probability that at least one of k trials succeeds; it rises with k. pass^k is the probability that all k trials succeed; it falls with k. The arithmetic from Anthropic's guide: an agent with a 75% per-trial success rate passes all three of three trials with probability 0.75³ ≈ 42%.

τ-bench introduced pass^k precisely because of this gap. In the 2024 paper, even state-of-the-art function-calling agents such as gpt-4o succeeded on fewer than half of the tasks, and pass^8 in the retail domain was below 25%. A coding agent can live with pass@k, because you only need one working patch. A support agent cannot: every customer gets one trial. Run each task at least three to five times and report both numbers.

Isolated environments, or the eval lies

Outcome grading needs a real environment: a seeded database, a sandbox for file and shell tools, mocked external APIs. Anthropic stresses that each trial must start from a clean state, because shared state causes correlated failures and can inflate scores. Their example: in internal evals Claude gained an unfair advantage on some tasks by reading the git history left by previous trials.

• Seed per trial. Spin up the database from a fixture, run, diff against the goal state, throw it away.
• Freeze time and randomness where tools depend on them ("orders from last week").
• Mock the outside world deterministically, but keep the error responses real: timeouts, 429s and validation errors are part of what you are testing.
• Keep the harness identical to production: same system prompt, same tool descriptions, same limits. Anthropic's point is that you evaluate the model and the harness together.

LLM judges for trajectories

Some qualities only a model can grade at scale: did the agent ask a sensible clarifying question, did it explain the refusal politely, was the plan reasonable. agentevals ships create_trajectory_llm_as_judge with a default trajectory-accuracy prompt; most eval frameworks have an equivalent. The rules for using one are the same as for any judge, only stricter because transcripts are long:

• One rubric item per dimension, each with a concrete pass and fail description.
• Give the judge a way out. Anthropic recommends an explicit "Unknown" answer when the transcript lacks the information, to stop the judge from guessing.
• Use partial credit for multi-step tasks. A support agent that identified the problem and verified the customer but failed the refund is better than one that failed immediately, and your scores should show that.
• Calibrate against humans on a sample before trusting the judge, and re-check when you change the judge model.

Capability suites, regression suites and where tasks come from

Anthropic separates two kinds of suites. Capability evals ask what the agent can do and should start with a low pass rate, giving you a hill to climb. Regression evals ask whether it still does what it used to and should sit near 100%. As capability tasks become reliable, they graduate into the regression suite. When a capability suite approaches 100% it stops telling you anything; their example is SWE-bench Verified, where scores started around 30% and frontier models now exceed 80%.

Where tasks come from matters more than how many you have. Anthropic's advice is to start early with 20 to 50 simple tasks drawn from real failures, because early changes have large effects and small samples detect them. My sources, in order of value: production traces where something went wrong (from your observability pipeline), the manual checks you already run before a release, and support tickets. Synthetic tasks fill gaps; they should not be the base.

Tooling: where the traces and graders live

You need three things: a way to capture full traces, a runner that executes tasks against an isolated environment several times, and a place to store grades over time. OpenAI's platform offers trace grading, scoring agent traces in its dashboard and turning them into eval runs. Open-source frameworks (agentevals with pytest or Vitest integration, DeepEval, Ragas, Promptfoo) run in CI. Observability platforms such as Langfuse connect production traces to datasets. The choice matters less than the habit: every agent change runs the regression suite before it ships, and the agent loop's budgets are part of what you assert.

Checklist

1. Grade the outcome against the environment state, not the final message.
2. Express the few must-hold rules as code invariants over the trajectory.
3. Use reference trajectories with the loosest match mode that still catches the failure (superset or subset before strict).
4. Run every task three to five times; report pass@1 and pass^k.
5. Start each trial from a clean, seeded environment that mirrors production.
6. Use LLM judges with one rubric item per dimension, an Unknown option and partial credit, calibrated against humans.
7. Keep a capability suite and a regression suite; graduate tasks between them.
8. Source tasks from real failures, and read failed transcripts every week.

For the retrieval side of agents that search documents, evaluating RAG covers the metrics. If you want help setting up agent evals for your team, see AI engineering.

Sources

1. Anthropic: Demystifying evals for AI agents (Jan 2026)
2. Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)
3. Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)
4. LangChain: agentevals (GitHub)
5. DeepEval docs: Tool Correctness
6. Ragas docs: Agentic or tool use metrics
7. OpenAI API docs: Trace grading