[{"data":1,"prerenderedAt":791},["ShallowReactive",2],{"blog-evaluating-ai-agents-trajectories-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":27,"sources":37,"cover":59,"og":60,"expertise":61,"locales":62,"lang":63,"title":66,"description":67,"coverAlt":68,"metaTitle":69,"takeaways":70,"faq":76,"toc":92,"blocks":126,"others":533},"evaluating-ai-agents-trajectories","2026-10-16",9,"llmops",[9,10,11,12,13],"Agent evals","Trajectories","Tool calling","pass^k","LLM-as-judge",[15,16,17,18,19,20,21,22,23,24,25,26],"evaluating AI agents","agent trajectory evaluation","agent evals","tool call accuracy","trajectory match evaluator","pass^k metric","tau-bench","LLM as a judge for agents","agent regression testing","outcome vs trajectory evaluation","agentevals","tool correctness metric",[28,31,34],{"name":29,"url":30},"Intelligent agent","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIntelligent_agent",{"name":32,"url":33},"Software testing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSoftware_testing",{"name":35,"url":36},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[38,41,44,47,50,53,56],{"title":39,"url":40},"Anthropic: Demystifying evals for AI agents (Jan 2026)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents",{"title":42,"url":43},"Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045",{"title":45,"url":46},"Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07982",{"title":48,"url":49},"LangChain: agentevals (GitHub)","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals",{"title":51,"url":52},"DeepEval docs: Tool Correctness","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fmetrics-tool-correctness",{"title":54,"url":55},"Ragas docs: Agentic or tool use metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002Fagents\u002F",{"title":57,"url":58},"OpenAI API docs: Trace grading","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ftrace-grading","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fog.jpg","ai-engineer",[63,64,65],"en","de","hu","Evaluating AI agents: grading trajectories, outcomes and reliability","How to evaluate AI agents: outcome vs trajectory graders, trajectory match modes, pass^k, isolated environments and LLM judges that hold up.","An agent trajectory of tool calls graded twice: a trajectory grader checks order and arguments, an outcome grader compares the database with the goal state.","Evaluating AI agents: trajectories, outcomes · Balázs Csorba",[71,72,73,74,75],"Grade the outcome against the environment state; the agent's final message is the weakest signal of success.","Use trajectory checks for invariants that must hold on every path, and avoid demanding one exact path, since agents find valid alternatives.","agentevals offers strict, unordered, subset and superset trajectory matching plus per-tool argument rules; DeepEval and Ragas cover similar ground.","pass^k, the chance that all k trials succeed, exposes unreliability that pass@k hides; at 75% per trial, three out of three is about 42%.","Start each trial in a clean environment, calibrate LLM judges against humans, and build tasks from 20 to 50 real failures.",[77,80,83,86,89],{"q":78,"a":79},"What is trajectory evaluation for AI agents?","Trajectory evaluation grades the sequence of steps an agent took: which tools it called, in which order and with which arguments. It complements outcome evaluation, which checks the final state of the environment, such as whether a refund record exists. Use trajectories mainly for rules that must hold on every path.",{"q":81,"a":82},"What is the difference between pass@k and pass^k?","pass@k is the probability that at least one of k attempts succeeds, so it rises as k grows. pass^k is the probability that all k attempts succeed, so it falls. Coding agents can use pass@k because one working patch is enough; customer-facing agents should be judged on pass^k because every customer gets one attempt.",{"q":84,"a":85},"Should an agent eval require an exact sequence of tool calls?","Only where the order is the rule, such as verifying a customer before a refund. Otherwise use looser modes like superset or subset, or grade the outcome. Anthropic reports a case where a model failed an eval as written by finding a policy loophole that produced a better result for the user.",{"q":87,"a":88},"How many tasks do I need to start evaluating an agent?","Anthropic suggests 20 to 50 simple tasks drawn from real failures as a good start, because early changes have large effects that small samples detect. Run each task several times and grow the suite as the agent matures.",{"q":90,"a":91},"Can I use an LLM as a judge for agent transcripts?","Yes, for qualities code cannot check, such as whether a clarifying question was sensible. Use one rubric item per dimension, give the judge an Unknown option, award partial credit on multi-step tasks and calibrate it against human grades on a sample.",[93,96,99,102,105,108,111,114,117,120,123],{"id":94,"title":95},"outcome-vs-trajectory","Outcome and trajectory: the two things you grade",{"id":97,"title":98},"three-kinds-of-graders","Code, model and human graders",{"id":100,"title":101},"trajectory-matching","Trajectory matching: strict, unordered, subset, superset",{"id":103,"title":104},"grade-outcomes-not-paths","Grade outcomes first, paths only for invariants",{"id":106,"title":107},"pass-at-k-vs-pass-hat-k","pass@k and pass^k: one trial is not a measurement",{"id":109,"title":110},"isolated-environments","Isolated environments, or the eval lies",{"id":112,"title":113},"llm-judges-for-trajectories","LLM judges for trajectories",{"id":115,"title":116},"capability-and-regression-suites","Capability suites, regression suites and where tasks come from",{"id":118,"title":119},"tooling","Tooling: where the traces and graders live",{"id":121,"title":122},"checklist","Checklist",{"id":124,"title":125},"sources","Sources",[127,140,149,152,174,177,186,187,190,228,231,232,244,277,300,302,331,332,335,342,344,352,353,363,370,373,374,377,401,402,409,431,432,443,451,458,459,477,478,497,509,510],{"type":128,"content":129},"paragraph",[130,131,135,136,139],"An agent that says \"your refund has been processed\" and an agent that actually processed the refund look identical in a chat log. Evaluating agents means looking past the final message: at the ",{"tag":132,"children":133},"strong",[134],"outcome"," (what changed in the world) and at the ",{"tag":132,"children":137},[138],"trajectory"," (which tools were called, in which order, with which arguments). Grading only the final text is how agents pass evals and fail customers.",{"type":128,"content":141},[142,143,148],"This article is the agent-specific follow-up to ",{"tag":144,"to":145,"children":146},"link","\u002Fblog\u002Fllm-evals-for-product-features",[147],"LLM evals for product features",". It covers the vocabulary, the three kinds of graders, trajectory matching with agentevals, DeepEval and Ragas, why pass^k matters more than pass@1 for customer-facing agents, and how to keep the eval environment from lying to you.",{"type":150,"level":151,"id":94,"text":95},"heading",2,{"type":128,"content":153},[154,155,158,159,162,163,166,167,170,171,173],"Anthropic's January 2026 guide gives a vocabulary worth adopting. A ",{"tag":132,"children":156},[157],"task"," is one test case with inputs and success criteria; each attempt is a ",{"tag":132,"children":160},[161],"trial",". A ",{"tag":132,"children":164},[165],"grader"," scores some aspect of a trial. The ",{"tag":132,"children":168},[169],"transcript"," (also called trace or trajectory) is the complete record: outputs, tool calls, reasoning, intermediate results. The ",{"tag":132,"children":172},[134]," is the final state of the environment. Their example: a flight-booking agent may say \"Your flight has been booked\", but the outcome is whether a reservation exists in the database.",{"type":128,"content":175},[176],"τ-bench, one of the reference agent benchmarks, grades exactly this way: it compares the database state at the end of a conversation with an annotated goal state. That is the strongest single check you can build, because it does not care what the agent said, only what it did.",{"type":178,"attrs":179,"inner":183,"caption":184},"diagram",{"viewBox":180,"role":181,"aria-labelledby":182},"0 0 720 300","img","tr-graders-t tr-graders-d","\u003Ctitle id=\"tr-graders-t\">Trajectory graders and outcome graders\u003C\u002Ftitle>\u003Cdesc id=\"tr-graders-d\">A trial runs from the task through a sequence of tool calls to a final message. Trajectory graders read the sequence of tool calls and arguments. Outcome graders ignore the conversation and compare the environment state at the end with the goal state. The final message alone is the weakest signal.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"ah-traj\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"10\" y=\"120\" width=\"132\" height=\"60\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"76\" y=\"146\" text-anchor=\"middle\" class=\"d-text\">task\u003C\u002Ftext>\u003Ctext x=\"76\" y=\"166\" text-anchor=\"middle\" class=\"d-small\">user request\u003C\u002Ftext>\u003Crect x=\"152\" y=\"120\" width=\"132\" height=\"60\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"218\" y=\"146\" text-anchor=\"middle\" class=\"d-text\">tool call 1\u003C\u002Ftext>\u003Ctext x=\"218\" y=\"166\" text-anchor=\"middle\" class=\"d-small\">lookup_order\u003C\u002Ftext>\u003Crect x=\"294\" y=\"120\" width=\"132\" height=\"60\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"360\" y=\"146\" text-anchor=\"middle\" class=\"d-text\">tool call 2\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"166\" text-anchor=\"middle\" class=\"d-small\">verify_customer\u003C\u002Ftext>\u003Crect x=\"436\" y=\"120\" width=\"132\" height=\"60\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"502\" y=\"146\" text-anchor=\"middle\" class=\"d-text\">tool call 3\u003C\u002Ftext>\u003Ctext x=\"502\" y=\"166\" text-anchor=\"middle\" class=\"d-small\">issue_refund\u003C\u002Ftext>\u003Crect x=\"578\" y=\"120\" width=\"132\" height=\"60\" rx=\"10\" class=\"d-fill-muted\" \u002F>\u003Ctext x=\"644\" y=\"146\" text-anchor=\"middle\" class=\"d-text\">final message\u003C\u002Ftext>\u003Ctext x=\"644\" y=\"166\" text-anchor=\"middle\" class=\"d-small\">\"refund done\"\u003C\u002Ftext>\u003Cpath d=\"M142 150H160\" class=\"d-line\" marker-end=\"url(#ah-traj)\" \u002F>\u003Cpath d=\"M284 150H302\" class=\"d-line\" marker-end=\"url(#ah-traj)\" \u002F>\u003Cpath d=\"M426 150H444\" class=\"d-line\" marker-end=\"url(#ah-traj)\" \u002F>\u003Cpath d=\"M568 150H586\" class=\"d-line\" marker-end=\"url(#ah-traj)\" \u002F>\u003Crect x=\"152\" y=\"20\" width=\"416\" height=\"56\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"360\" y=\"44\" text-anchor=\"middle\" class=\"d-text\">trajectory grader\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"64\" text-anchor=\"middle\" class=\"d-small\">order, args, invariants\u003C\u002Ftext>\u003Cpath d=\"M218 118V78\" class=\"d-line d-dash\" marker-end=\"url(#ah-traj)\" \u002F>\u003Cpath d=\"M360 118V78\" class=\"d-line d-dash\" marker-end=\"url(#ah-traj)\" \u002F>\u003Cpath d=\"M502 118V78\" class=\"d-line d-dash\" marker-end=\"url(#ah-traj)\" \u002F>\u003Crect x=\"294\" y=\"230\" width=\"274\" height=\"56\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"431\" y=\"254\" text-anchor=\"middle\" class=\"d-text\">environment\u003C\u002Ftext>\u003Ctext x=\"431\" y=\"274\" text-anchor=\"middle\" class=\"d-small\">orders, refunds tables\u003C\u002Ftext>\u003Cpath d=\"M502 180V228\" class=\"d-line-accent\" marker-end=\"url(#ah-traj)\" \u002F>\u003Crect x=\"20\" y=\"230\" width=\"230\" height=\"56\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"135\" y=\"254\" text-anchor=\"middle\" class=\"d-text\">outcome grader\u003C\u002Ftext>\u003Ctext x=\"135\" y=\"274\" text-anchor=\"middle\" class=\"d-small\">DB state = goal state\u003C\u002Ftext>\u003Cpath d=\"M294 258H252\" class=\"d-line-accent\" marker-end=\"url(#ah-traj)\" \u002F>",[185],"Grade the trajectory for rules that must hold on the way, and the outcome for what actually changed. The final message is the least reliable of the three.",{"type":150,"level":151,"id":97,"text":98},{"type":128,"content":188},[189],"Each grader looks at either the transcript or the outcome. Anthropic's guide splits them into three families, and a good agent eval mixes all three:",{"type":191,"head":192,"rows":201},"table",[193,195,197,199],[194],"Grader",[196],"Typical checks",[198],"Strengths",[200],"Weaknesses",[202,212,220],[203,206,208,210],[204,205],"tag","children",[207],"Outcome verification, tool calls used and their parameters, turns taken, token usage, string and regex matches, tests",[209],"Fast, cheap, objective, reproducible",[211],"Brittle to valid variations; little nuance",[213,214,216,218],[204,205],[215],"Rubric scoring, natural-language assertions, pairwise comparison, reference-based grading",[217],"Flexible, handles open-ended output",[219],"Non-deterministic, costs more, must be calibrated against humans",[221,222,224,226],[204,205],[223],"Expert review, spot-check sampling, A\u002FB tests",[225],"Gold standard, matches expert judgment",[227],"Slow and expensive; use it to calibrate the others",{"type":128,"content":229},[230],"Scores can be combined as a weighted sum against a threshold, as a binary all-must-pass, or as a hybrid. My default for business agents: binary on safety-relevant code checks (no refund without verification, ever), weighted on everything else.",{"type":150,"level":151,"id":100,"text":101},{"type":128,"content":233},[234,235,238,239,243],"When you have a reference trajectory, the question is how exactly the agent has to follow it. LangChain's open-source ",{"tag":236,"href":49,"children":237},"a",[25]," package names the four useful answers as ",{"tag":240,"children":241},"code",[242],"trajectory_match_mode"," values:",{"type":191,"head":245,"rows":252},[246,248,250],[247],"Mode",[249],"Passes when",[251],"Use it for",[253,259,265,271],[254,255,257],[204,205],[256],"Same messages, same order, same tool calls (message text may differ)",[258],"Workflows where order is the rule: verify, then refund",[260,261,263],[204,205],[262],"Same tool calls in any order",[264],"Information gathering where every source must be consulted",[266,267,269],[204,205],[268],"The agent called no tools beyond the reference",[270],"Guarding against extra, expensive or risky calls",[272,273,275],[204,205],[274],"The agent called at least the reference tools",[276],"Minimum required steps, extra lookups allowed",{"type":128,"content":278},[279,280,283,284,287,288,291,292,295,296,299],"Arguments are matched separately. By default agentevals requires identical arguments; ",{"tag":240,"children":281},[282],"tool_args_match_mode"," can be set to ",{"tag":240,"children":285},[286],"ignore",", ",{"tag":240,"children":289},[290],"subset"," or ",{"tag":240,"children":293},[294],"superset",", and ",{"tag":240,"children":297},[298],"tool_args_match_overrides"," takes per-tool rules, including a custom comparator (case-insensitive city names, for example):",{"type":240,"code":301},"from agentevals.trajectory.match import create_trajectory_match_evaluator\n\n# Refund flow: the agent may look things up freely, but must make these calls.\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"superset\",\n    tool_args_match_overrides={\n        \"issue_refund\": [\"order_id\"],          # only order_id must match exactly\n        \"lookup_order\": \"ignore\",              # any lookup arguments are fine\n    },\n)\nresult = evaluator(outputs=agent_messages, reference_outputs=reference_messages)\n# {'key': 'trajectory_superset_match', 'score': True, 'comment': None}",{"type":128,"content":303},[304,305,308,309,312,313,316,317,320,321,325,326,330],"Other frameworks cover the same ground with different names. DeepEval's ",{"tag":240,"children":306},[307],"ToolCorrectnessMetric"," compares expected and called tools, by name by default, optionally also by input parameters and output, with flags for ordering and exact match. Ragas offers ",{"tag":240,"children":310},[311],"ToolCallAccuracy"," (strict order by default, with a flexible mode), ",{"tag":240,"children":314},[315],"ToolCallF1"," for precision and recall over calls, and ",{"tag":240,"children":318},[319],"AgentGoalAccuracy"," for when only the outcome matters. Pick one framework per project; the concepts transfer. My reviews of ",{"tag":144,"to":322,"children":323},"\u002Ftools\u002Fdeepeval",[324],"DeepEval"," and ",{"tag":144,"to":327,"children":328},"\u002Ftools\u002Fragas",[329],"Ragas"," cover the trade-offs.",{"type":150,"level":151,"id":103,"text":104},{"type":128,"content":333},[334],"The temptation with trajectories is to demand the exact path you would have taken. Anthropic warns against it: agents regularly find valid approaches the eval designer did not anticipate, so \"it's often better to grade what the agent produced, not the path it took.\" Their own example is striking. Opus 4.5 solved a τ²-bench flight-booking problem by finding a loophole in the policy; it \"failed\" the eval as written while producing a better result for the user.",{"type":128,"content":336},[337,338,341],"So I use trajectory checks for ",{"tag":132,"children":339},[340],"invariants",", the few rules that must hold on every path, and outcome checks for success. Invariants are usually short and easy to express in code:",{"type":240,"code":343},"def check_invariants(calls: list[dict]) -> list[str]:\n    \"\"\"calls: tool calls from the transcript, in order: {\"name\": ..., \"args\": {...}}\"\"\"\n    errors, verified = [], False\n    for c in calls:\n        if c[\"name\"] == \"verify_customer\":\n            verified = True\n        if c[\"name\"] == \"issue_refund\":\n            if not verified:\n                errors.append(\"refund before customer verification\")\n            if c[\"args\"][\"amount\"] > 500:\n                errors.append(\"refund above approval limit\")\n        if c[\"name\"] in {\"delete_account\", \"send_bulk_email\"}:\n            errors.append(f\"forbidden tool: {c['name']}\")\n    if len(calls) > 25:\n        errors.append(f\"{len(calls)} tool calls, budget is 25\")\n    return errors",{"type":128,"content":345},[346,347,351],"This is binary and cheap, it runs on every trial, and it does not punish creative paths. Combine it with the outcome check against the database, and you have caught the two failures that matter most: the agent did the wrong thing, or it did the right thing in a forbidden way. For deciding which actions need a human gate in the first place, see ",{"tag":144,"to":348,"children":349},"\u002Fblog\u002Fhuman-in-the-loop-ai-agents",[350],"human in the loop for AI agents",".",{"type":150,"level":151,"id":106,"text":107},{"type":128,"content":354},[355,356,359,360,362],"Agents are non-deterministic, so a task that passes once may fail on the next run. Two metrics describe this. ",{"tag":132,"children":357},[358],"pass@k"," is the probability that at least one of k trials succeeds; it rises with k. ",{"tag":132,"children":361},[12]," is the probability that all k trials succeed; it falls with k. The arithmetic from Anthropic's guide: an agent with a 75% per-trial success rate passes all three of three trials with probability 0.75³ ≈ 42%.",{"type":178,"attrs":364,"inner":367,"caption":368},{"viewBox":365,"role":181,"aria-labelledby":366},"0 0 720 290","pk-chart-t pk-chart-d","\u003Ctitle id=\"pk-chart-t\">pass@k and pass^k for a 75% per-trial success rate\u003C\u002Ftitle>\u003Cdesc id=\"pk-chart-d\">Bar chart. At k equals 1 both metrics are 75 percent. At k equals 3 pass@k is 98 percent and pass^k is 42 percent. At k equals 10 pass@k is above 99.9 percent and pass^k is 6 percent.\u003C\u002Fdesc>\u003Cpath d=\"M60 250H700\" class=\"d-line\" \u002F>\u003Crect x=\"110\" y=\"100\" width=\"56\" height=\"150\" class=\"d-fill-accent\" \u002F>\u003Crect x=\"174\" y=\"100\" width=\"56\" height=\"150\" class=\"d-fill-muted\" \u002F>\u003Ctext x=\"138\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">75%\u003C\u002Ftext>\u003Ctext x=\"202\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">75%\u003C\u002Ftext>\u003Ctext x=\"170\" y=\"274\" text-anchor=\"middle\" class=\"d-label\">k = 1\u003C\u002Ftext>\u003Crect x=\"310\" y=\"53\" width=\"56\" height=\"197\" class=\"d-fill-accent\" \u002F>\u003Crect x=\"374\" y=\"166\" width=\"56\" height=\"84\" class=\"d-fill-muted\" \u002F>\u003Ctext x=\"338\" y=\"45\" text-anchor=\"middle\" class=\"d-small\">98%\u003C\u002Ftext>\u003Ctext x=\"402\" y=\"158\" text-anchor=\"middle\" class=\"d-small\">42%\u003C\u002Ftext>\u003Ctext x=\"370\" y=\"274\" text-anchor=\"middle\" class=\"d-label\">k = 3\u003C\u002Ftext>\u003Crect x=\"510\" y=\"50\" width=\"56\" height=\"200\" class=\"d-fill-accent\" \u002F>\u003Crect x=\"574\" y=\"239\" width=\"56\" height=\"11\" class=\"d-fill-muted\" \u002F>\u003Ctext x=\"538\" y=\"42\" text-anchor=\"middle\" class=\"d-small\">>99.9%\u003C\u002Ftext>\u003Ctext x=\"602\" y=\"231\" text-anchor=\"middle\" class=\"d-small\">6%\u003C\u002Ftext>\u003Ctext x=\"570\" y=\"274\" text-anchor=\"middle\" class=\"d-label\">k = 10\u003C\u002Ftext>\u003Crect x=\"70\" y=\"8\" width=\"14\" height=\"14\" class=\"d-fill-accent\" \u002F>\u003Ctext x=\"92\" y=\"20\" text-anchor=\"start\" class=\"d-small\">pass@k (at least one)\u003C\u002Ftext>\u003Crect x=\"70\" y=\"28\" width=\"14\" height=\"14\" class=\"d-fill-muted\" \u002F>\u003Ctext x=\"92\" y=\"40\" text-anchor=\"start\" class=\"d-small\">pass^k (all of them)\u003C\u002Ftext>",[369],"The same agent looks almost perfect under pass@k and unreliable under pass^k. Customer-facing agents should be judged on the second.",{"type":128,"content":371},[372],"τ-bench introduced pass^k precisely because of this gap. In the 2024 paper, even state-of-the-art function-calling agents such as gpt-4o succeeded on fewer than half of the tasks, and pass^8 in the retail domain was below 25%. A coding agent can live with pass@k, because you only need one working patch. A support agent cannot: every customer gets one trial. Run each task at least three to five times and report both numbers.",{"type":150,"level":151,"id":109,"text":110},{"type":128,"content":375},[376],"Outcome grading needs a real environment: a seeded database, a sandbox for file and shell tools, mocked external APIs. Anthropic stresses that each trial must start from a clean state, because shared state causes correlated failures and can inflate scores. Their example: in internal evals Claude gained an unfair advantage on some tasks by reading the git history left by previous trials.",{"type":378,"ordered":379,"items":380},"list",false,[381,386,391,396],[382,385],{"tag":132,"children":383},[384],"Seed per trial."," Spin up the database from a fixture, run, diff against the goal state, throw it away.",[387,390],{"tag":132,"children":388},[389],"Freeze time and randomness"," where tools depend on them (\"orders from last week\").",[392,395],{"tag":132,"children":393},[394],"Mock the outside world deterministically,"," but keep the error responses real: timeouts, 429s and validation errors are part of what you are testing.",[397,400],{"tag":132,"children":398},[399],"Keep the harness identical to production:"," same system prompt, same tool descriptions, same limits. Anthropic's point is that you evaluate the model and the harness together.",{"type":150,"level":151,"id":112,"text":113},{"type":128,"content":403},[404,405,408],"Some qualities only a model can grade at scale: did the agent ask a sensible clarifying question, did it explain the refusal politely, was the plan reasonable. agentevals ships ",{"tag":240,"children":406},[407],"create_trajectory_llm_as_judge"," with a default trajectory-accuracy prompt; most eval frameworks have an equivalent. The rules for using one are the same as for any judge, only stricter because transcripts are long:",{"type":378,"ordered":379,"items":410},[411,416,421,426],[412,415],{"tag":132,"children":413},[414],"One rubric item per dimension,"," each with a concrete pass and fail description.",[417,420],{"tag":132,"children":418},[419],"Give the judge a way out."," Anthropic recommends an explicit \"Unknown\" answer when the transcript lacks the information, to stop the judge from guessing.",[422,425],{"tag":132,"children":423},[424],"Use partial credit for multi-step tasks."," A support agent that identified the problem and verified the customer but failed the refund is better than one that failed immediately, and your scores should show that.",[427,430],{"tag":132,"children":428},[429],"Calibrate against humans"," on a sample before trusting the judge, and re-check when you change the judge model.",{"type":150,"level":151,"id":115,"text":116},{"type":128,"content":433},[434,435,438,439,442],"Anthropic separates two kinds of suites. ",{"tag":132,"children":436},[437],"Capability evals"," ask what the agent can do and should start with a low pass rate, giving you a hill to climb. ",{"tag":132,"children":440},[441],"Regression evals"," ask whether it still does what it used to and should sit near 100%. As capability tasks become reliable, they graduate into the regression suite. When a capability suite approaches 100% it stops telling you anything; their example is SWE-bench Verified, where scores started around 30% and frontier models now exceed 80%.",{"type":128,"content":444},[445,446,450],"Where tasks come from matters more than how many you have. Anthropic's advice is to start early with 20 to 50 simple tasks drawn from real failures, because early changes have large effects and small samples detect them. My sources, in order of value: production traces where something went wrong (from your ",{"tag":144,"to":447,"children":448},"\u002Fblog\u002Fagent-observability-opentelemetry",[449],"observability pipeline","), the manual checks you already run before a release, and support tickets. Synthetic tasks fill gaps; they should not be the base.",{"type":452,"variant":453,"title":454,"body":455},"callout","tip","Read the transcripts",[456],[457],"No metric replaces reading failed trials. Anthropic calls it a critical skill: a failing transcript tells you whether the agent made a real mistake or your grader rejected a valid solution. Budget an hour per week for it, and fix graders as often as you fix prompts.",{"type":150,"level":151,"id":118,"text":119},{"type":128,"content":460},[461,462,466,467,471,472,476],"You need three things: a way to capture full traces, a runner that executes tasks against an isolated environment several times, and a place to store grades over time. OpenAI's platform offers trace grading, scoring agent traces in its dashboard and turning them into eval runs. Open-source frameworks (agentevals with pytest or Vitest integration, DeepEval, Ragas, ",{"tag":144,"to":463,"children":464},"\u002Ftools\u002Fpromptfoo",[465],"Promptfoo",") run in CI. Observability platforms such as ",{"tag":144,"to":468,"children":469},"\u002Ftools\u002Flangfuse",[470],"Langfuse"," connect production traces to datasets. The choice matters less than the habit: every agent change runs the regression suite before it ships, and the ",{"tag":144,"to":473,"children":474},"\u002Fblog\u002Fagent-loop-explained",[475],"agent loop's"," budgets are part of what you assert.",{"type":150,"level":151,"id":121,"text":122},{"type":378,"ordered":479,"items":480},true,[481,483,485,487,489,491,493,495],[482],"Grade the outcome against the environment state, not the final message.",[484],"Express the few must-hold rules as code invariants over the trajectory.",[486],"Use reference trajectories with the loosest match mode that still catches the failure (superset or subset before strict).",[488],"Run every task three to five times; report pass@1 and pass^k.",[490],"Start each trial from a clean, seeded environment that mirrors production.",[492],"Use LLM judges with one rubric item per dimension, an Unknown option and partial credit, calibrated against humans.",[494],"Keep a capability suite and a regression suite; graduate tasks between them.",[496],"Source tasks from real failures, and read failed transcripts every week.",{"type":128,"content":498},[499,500,504,505,351],"For the retrieval side of agents that search documents, ",{"tag":144,"to":501,"children":502},"\u002Fblog\u002Frag-evaluation-metrics",[503],"evaluating RAG"," covers the metrics. If you want help setting up agent evals for your team, see ",{"tag":144,"to":506,"children":507},"\u002Fexpertise\u002Fai-engineer",[508],"AI engineering",{"type":150,"level":151,"id":124,"text":125},{"type":378,"ordered":479,"items":511},[512,515,518,521,524,527,530],[513],{"tag":236,"href":40,"children":514},[39],[516],{"tag":236,"href":43,"children":517},[42],[519],{"tag":236,"href":46,"children":520},[45],[522],{"tag":236,"href":49,"children":523},[48],[525],{"tag":236,"href":52,"children":526},[51],[528],{"tag":236,"href":55,"children":529},[54],[531],{"tag":236,"href":58,"children":532},[57],[534,586,640,687],{"slug":535,"published":5,"minutes":6,"category":7,"tags":536,"keywords":542,"about":554,"sources":561,"cover":580,"og":581,"expertise":61,"locales":582,"lang":63,"title":583,"description":584,"coverAlt":585},"structured-outputs-json-schema-llm",[537,538,539,540,541],"Structured outputs","JSON Schema","Constrained decoding","LLM APIs","Validation",[543,544,545,546,547,548,549,550,551,552,553],"structured outputs LLM","JSON schema LLM output","OpenAI structured outputs strict mode","Claude structured outputs","Gemini structured output","constrained decoding","JSON mode vs structured outputs","validate LLM JSON output","Zod structured output","Pydantic LLM output","vLLM structured outputs",[555,558,560],{"name":556,"url":557},"JSON","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FJSON",{"name":538,"url":559},"https:\u002F\u002Fjson-schema.org\u002F",{"name":35,"url":36},[562,565,568,571,574,577],{"title":563,"url":564},"OpenAI API docs: Structured model outputs","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fstructured-outputs",{"title":566,"url":567},"Claude API docs: Structured outputs","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fstructured-outputs",{"title":569,"url":570},"Gemini API docs: Structured outputs","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fstructured-output",{"title":572,"url":573},"vLLM docs: Structured Outputs","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fstructured_outputs.html",{"title":575,"url":576},"Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.02442",{"title":578,"url":579},"Geng et al.: JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.10868","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fog.jpg",[63,64,65],"Structured outputs with JSON Schema: what the guarantee covers, and what it does not","How structured outputs and constrained decoding work on OpenAI, Claude, Gemini and vLLM, which schema rules matter, and what you still validate.","A JSON Schema compiled into a grammar that masks tokens, producing schema-valid JSON that then passes a business validation step.",{"slug":587,"published":5,"minutes":588,"category":7,"tags":589,"keywords":595,"about":607,"sources":614,"cover":634,"og":635,"expertise":61,"locales":636,"lang":63,"title":637,"description":638,"coverAlt":639},"llm-model-migration-regression-testing",10,[590,591,592,593,594],"Model migration","Regression testing","Deprecations","LLM evals","Canary releases",[596,597,598,599,600,601,602,603,604,605,606],"LLM model migration","LLM regression testing","model deprecation OpenAI","Claude model deprecation","switch LLM model without breaking","LLM model upgrade checklist","prompt regression testing","shadow testing LLM","model retirement date","golden dataset LLM","LLM drift monitoring",[608,610,611],{"name":591,"url":609},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRegression_testing",{"name":35,"url":36},{"name":612,"url":613},"Feature toggle","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FFeature_toggle",[615,618,621,624,627,630,633],{"title":616,"url":617},"Claude API docs: Model deprecations","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fabout-claude\u002Fmodel-deprecations",{"title":619,"url":620},"OpenAI API docs: Deprecations","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fdeprecations",{"title":622,"url":623},"Gemini API docs: Models and version name patterns","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fmodels",{"title":625,"url":626},"Claude API docs: Claude Opus 5.5 migration guide","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fmodels\u002Fopus-5-5\u002Fmigration-guide",{"title":628,"url":629},"Chen, Zaharia, Zou: How is ChatGPT's behavior changing over time? (2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.09009",{"title":631,"url":632},"Sclar et al.: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.11324",{"title":39,"url":40},"\u002Fimages\u002Fblog\u002Fllm-model-migration-regression-testing\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-model-migration-regression-testing\u002Fog.jpg",[63,64,65],"LLM model migrations without regressions: deprecations, evals, shadow traffic","A playbook for switching LLM models: deprecation timelines, contract fixes, a regression suite, prompt re-tuning, shadow traffic and canary rollout.","Five migration stages from inventory and contract fixes through offline evals and shadow traffic to a canary rollout with a rollback path.",{"slug":641,"published":5,"minutes":642,"category":7,"tags":643,"keywords":649,"about":660,"sources":668,"cover":681,"og":682,"expertise":61,"locales":683,"lang":63,"title":684,"description":685,"coverAlt":686},"semantic-caching-llm-responses",6,[644,645,646,647,648],"Semantic cache","LLM cost","Latency","Embeddings","Redis",[650,651,652,653,654,655,656,657,658,659],"semantic caching llm","semantic cache","llm response cache","gptcache","semantic cache similarity threshold","prompt caching vs semantic caching","reduce llm api cost","cache llm answers redis","semantic cache false positive","llm cache invalidation",[661,664,667],{"name":662,"url":663},"Cache (computing)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCache_(computing)",{"name":665,"url":666},"Sentence embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSentence_embedding",{"name":35,"url":36},[669,672,675,678],{"title":670,"url":671},"Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2411.05276",{"title":673,"url":674},"Zilliz: GPTCache README (modules, similarity evaluation, eviction)","https:\u002F\u002Fgithub.com\u002Fzilliztech\u002FGPTCache",{"title":676,"url":677},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching",{"title":679,"url":680},"pgvector README: vector types and indexes","https:\u002F\u002Fgithub.com\u002Fpgvector\u002Fpgvector","\u002Fimages\u002Fblog\u002Fsemantic-caching-llm-responses\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fsemantic-caching-llm-responses\u002Fog.jpg",[63,64,65],"Semantic caching for LLM apps: when reusing answers is safe, and when it is not","How semantic caches reuse LLM answers for similar questions, how they differ from prompt caching, and how to avoid wrong hits, leaks and stale answers.","A new question is embedded and compared with cached questions; above a similarity threshold the stored answer is returned, below it the model is called.",{"slug":688,"published":689,"minutes":588,"category":7,"tags":690,"keywords":696,"about":705,"sources":715,"cover":785,"og":786,"expertise":61,"locales":787,"lang":63,"title":788,"description":789,"coverAlt":790},"self-hosted-llm-gdpr-cost","2026-10-09",[691,692,693,694,695],"Self-hosting","GDPR","vLLM","GPU cost","Open-weight models",[697,698,699,700,701,702,703,704],"self-hosted LLM GDPR","self-host LLM data protection","LLM GPU memory requirements","vLLM throughput benchmark","EU GPU cloud for LLMs","LLM API vs self-hosting break-even","open-weight LLM hardware cost","data processing agreement LLM",[706,709,710,713],{"name":707,"url":708},"General Data Protection Regulation","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGeneral_Data_Protection_Regulation",{"name":35,"url":36},{"name":711,"url":712},"Graphics processing unit","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGraphics_processing_unit",{"name":693,"url":714},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm",[716,719,722,725,728,731,734,737,740,743,746,749,752,755,758,761,764,767,770,773,776,779,782],{"title":717,"url":718},"openai\u002Fgpt-oss-120b model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fopenai\u002Fgpt-oss-120b",{"title":720,"url":721},"mistralai\u002FMistral-Small-3.2-24B-Instruct-2506 model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-Small-3.2-24B-Instruct-2506",{"title":723,"url":724},"meta-llama\u002FLlama-3.3-70B-Instruct model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FLlama-3.3-70B-Instruct",{"title":726,"url":727},"mistralai\u002FMistral-Large-3-675B-Instruct-2512 model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-Large-3-675B-Instruct-2512",{"title":729,"url":730},"vLLM blog: v0.6.0 performance update (September 2024)","https:\u002F\u002Fblog.vllm.ai\u002F2024\u002F09\u002F05\u002Fperf-update.html",{"title":732,"url":733},"vLLM documentation: engine arguments","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fconfiguration\u002Fengine_args.html",{"title":735,"url":736},"vLLM documentation: vllm bench serve","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fbench\u002Fserve.html",{"title":738,"url":739},"SemiAnalysis InferenceX: Llama 3.3 70B on H100 vs H200","https:\u002F\u002Finferencex.semianalysis.com\u002Fcompare\u002Fllama-3-3-70b-h100-vs-h200",{"title":741,"url":742},"Scaleway GPU instances pricing","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fpricing\u002Fgpu\u002F",{"title":744,"url":745},"Scaleway blog: a transparent update on Scaleway pricing (27 April 2026)","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fblog\u002Fa-transparent-update-on-scaleway-pricing\u002F",{"title":747,"url":748},"Scaleway Generative APIs pricing","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fpricing\u002Fmodel-as-a-service\u002F",{"title":750,"url":751},"OVHcloud public cloud price list","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fprices\u002F",{"title":753,"url":754},"OVHcloud AI Endpoints","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fai-endpoints\u002F",{"title":756,"url":757},"OVHcloud AI Endpoints model catalogue","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fai-endpoints\u002Fcatalog\u002F",{"title":759,"url":760},"Hetzner dedicated GPU servers","https:\u002F\u002Fwww.hetzner.com\u002Fdedicated-rootserver\u002Fmatrix-gpu\u002F",{"title":762,"url":763},"NVIDIA newsroom: GeForce RTX 50 Series launch","https:\u002F\u002Fnvidianews.nvidia.com\u002Fnews\u002Fnvidia-blackwell-geforce-rtx-50-series-opens-new-world-of-ai-computer-graphics",{"title":765,"url":766},"NVIDIA GeForce RTX 5090","https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fgeforce\u002Fgraphics-cards\u002F50-series\u002Frtx-5090\u002F",{"title":768,"url":769},"NVIDIA GeForce software licence","https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fdrivers\u002Fgeforce-license\u002F",{"title":771,"url":772},"GDPR Article 28: processor","https:\u002F\u002Fgdpr-info.eu\u002Fart-28-gdpr\u002F",{"title":774,"url":775},"GDPR Article 44: general principle for transfers","https:\u002F\u002Fgdpr-info.eu\u002Fart-44-gdpr\u002F",{"title":777,"url":778},"European Commission: EU-US data transfers (Data Privacy Framework)","https:\u002F\u002Fcommission.europa.eu\u002Flaw\u002Flaw-topic\u002Fdata-protection\u002Finternational-dimension-data-protection\u002Feu-us-data-transfers_en",{"title":780,"url":781},"EDPB Opinion 28\u002F2024 on AI models (adopted 17 December 2024)","https:\u002F\u002Fwww.edpb.europa.eu\u002Fsystem\u002Ffiles\u002F2024-12\u002Fedpb_opinion_202428_ai-models_en.pdf",{"title":783,"url":784},"Mistral AI pricing","https:\u002F\u002Fmistral.ai\u002Fpricing","\u002Fimages\u002Fblog\u002Fself-hosted-llm-gdpr-cost\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fself-hosted-llm-gdpr-cost\u002Fog.jpg",[63,64,65],"Self-hosting LLMs for GDPR: when it is required and what it costs","Self-hosting an LLM for GDPR: when it is required, GPU memory for open-weight models, EU prices as of October 2026 and break-even per million tokens.","Cover art for self-hosted LLMs under GDPR: a decision between your own GPUs and an EU API, with a break-even line.",1792177152110]