[{"data":1,"prerenderedAt":799},["ShallowReactive",2],{"blog-llm-model-migration-regression-testing-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":26,"sources":35,"cover":57,"og":58,"expertise":59,"locales":60,"lang":61,"title":64,"description":65,"coverAlt":66,"metaTitle":67,"takeaways":68,"faq":74,"toc":90,"blocks":124,"others":539},"llm-model-migration-regression-testing","2026-10-16",10,"llmops",[9,10,11,12,13],"Model migration","Regression testing","Deprecations","LLM evals","Canary releases",[15,16,17,18,19,20,21,22,23,24,25],"LLM model migration","LLM regression testing","model deprecation OpenAI","Claude model deprecation","switch LLM model without breaking","LLM model upgrade checklist","prompt regression testing","shadow testing LLM","model retirement date","golden dataset LLM","LLM drift monitoring",[27,29,32],{"name":10,"url":28},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRegression_testing",{"name":30,"url":31},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",{"name":33,"url":34},"Feature toggle","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FFeature_toggle",[36,39,42,45,48,51,54],{"title":37,"url":38},"Claude API docs: Model deprecations","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fabout-claude\u002Fmodel-deprecations",{"title":40,"url":41},"OpenAI API docs: Deprecations","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fdeprecations",{"title":43,"url":44},"Gemini API docs: Models and version name patterns","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fmodels",{"title":46,"url":47},"Claude API docs: Claude Opus 5.5 migration guide","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fmodels\u002Fopus-5-5\u002Fmigration-guide",{"title":49,"url":50},"Chen, Zaharia, Zou: How is ChatGPT's behavior changing over time? (2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.09009",{"title":52,"url":53},"Sclar et al.: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.11324",{"title":55,"url":56},"Anthropic: Demystifying evals for AI agents (Jan 2026)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents","\u002Fimages\u002Fblog\u002Fllm-model-migration-regression-testing\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-model-migration-regression-testing\u002Fog.jpg","ai-engineer",[61,62,63],"en","de","hu","LLM model migrations without regressions: deprecations, evals, shadow traffic","A playbook for switching LLM models: deprecation timelines, contract fixes, a regression suite, prompt re-tuning, shadow traffic and canary rollout.","Five migration stages from inventory and contract fixes through offline evals and shadow traffic to a canary rollout with a rollback path.","LLM model migration and regression testing · Balázs Csorba",[69,70,71,72,73],"Providers retire models on a schedule: OpenAI gives at least six months for GA models, Anthropic at least 60 days, and Gemini previews as little as two weeks.","A model swap changes three things: the API contract, behaviour on your prompts, and token counts and cost; check each separately.","Pin exact model IDs, centralise them per use case and inventory real usage from provider exports and gateway logs before testing.","Run old and new model side by side on a golden set from production, gate on success, parse failures, refusals, length, cost and latency, and read every loss.","Roll out through shadow traffic and a flagged canary, version prompt and model together, and keep the suite running weekly afterwards.",[75,78,81,84,87],{"q":76,"a":77},"How much notice do LLM providers give before retiring a model?","OpenAI documents at least six months for generally available models, three months for specialized variants and possibly two weeks for preview models. Anthropic gives at least 60 days for publicly released models. Google deprecates Gemini preview models with at least two weeks' notice and swaps the model behind -latest aliases.",{"q":79,"a":80},"Can I just change the model ID and redeploy?","Rarely. Newer models can reject parameters the old one accepted, such as disabled thinking, forced tool choice or prefills on Claude Opus 5.5, prompts tuned for one model behave differently on another, and token counts can change. Fix the contract first, then run a regression suite before rollout.",{"q":82,"a":83},"What should a regression test suite for an LLM migration measure?","Task success on a golden set built from production traffic and past incidents, schema and parse failures, refusal rate, output length, tokens and cost per completed task, and latency. Compare old and new model on the same inputs and read every case where the new model loses.",{"q":85,"a":86},"What is shadow testing for LLMs?","Shadow testing sends a copy of live requests to the new model and logs its answers next to the served ones without showing them to users. It finds cases your offline suite missed. Make sure the shadow path cannot trigger side effects, especially for agents with write tools.",{"q":88,"a":89},"Why do my prompts perform worse on a newer, better model?","Prompts accumulate workarounds for the old model's quirks, and research shows that the best prompt format correlates only weakly between models. Remove old workarounds first, re-tune reasoning or effort settings, use structured outputs instead of format coaxing, and only then add new instructions tied to failing cases.",[91,94,97,100,103,106,109,112,115,118,121],{"id":92,"title":93},"why-model-swaps-break","Why a model swap breaks things",{"id":95,"title":96},"deprecation-timelines","Deprecation timelines at OpenAI, Anthropic and Google",{"id":98,"title":99},"pin-and-inventory","Pin model IDs and know where they are used",{"id":101,"title":102},"migration-pipeline","The migration pipeline",{"id":104,"title":105},"contract-changes","Stage 2: fix the contract before you test behaviour",{"id":107,"title":108},"regression-suite","Stage 3: the regression suite for a model swap",{"id":110,"title":111},"retune-prompts","Re-tune prompts, effort and old workarounds",{"id":113,"title":114},"shadow-and-canary","Stages 4 and 5: shadow traffic and canary",{"id":116,"title":117},"continuous-regression","Keep the suite running after the migration",{"id":119,"title":120},"checklist","Migration checklist",{"id":122,"title":123},"sources","Sources",[125,129,132,135,138,163,166,167,170,207,214,215,218,252,253,256,265,266,269,336,343,344,347,398,401,403,416,417,420,442,449,450,457,469,470,478,486,487,507,514,515],{"type":126,"content":127},"paragraph",[128],"Every LLM feature you ship has an expiry date you did not choose. Providers retire models on a schedule, and the replacement is never a drop-in: request parameters get rejected, prompts that were tuned for the old model behave differently, and the tokenizer may count your inputs differently. If Claude Sonnet 4.5 is in your stack, the clock is visible right now: Anthropic deprecated it on 30 September 2026 and retires it on 30 November 2026, with Claude Sonnet 5.5 as the recommended replacement.",{"type":126,"content":130},[131],"This article is a migration playbook: how deprecation timelines work at OpenAI, Anthropic and Google, how to find every call site, what a regression suite for a model swap should measure, and how to roll the new model out with shadow traffic and a canary instead of a big-bang switch.",{"type":133,"level":134,"id":92,"text":93},"heading",2,{"type":126,"content":136},[137],"Three different things change when you move to a new model, and each needs its own check:",{"type":139,"ordered":140,"items":141},"list",false,[142,148,153],[143,147],{"tag":144,"children":145},"strong",[146],"The API contract."," Parameters that worked on the old model return a 400 on the new one. The Claude Opus 5.5 migration guide lists several: disabling thinking, thinking budgets, forced tool choice, non-default sampling parameters and assistant prefills are all rejected.",[149,152],{"tag":144,"children":150},[151],"Behaviour."," The same prompt produces different outputs. Sclar et al. measured accuracy differences of up to 76 points from meaning-preserving formatting changes alone, and found that which format works best only weakly correlates between models. A prompt tuned for model A is not tuned for model B.",[154,157,158,162],{"tag":144,"children":155},[156],"Cost and limits."," Anthropic's guide notes that the tokenizer introduced with Opus 4.7 may use roughly 1x to 1.35x as many tokens as earlier models for the same text. Your ",{"tag":159,"children":160},"code",[161],"max_tokens",", compaction triggers and cost estimates move with it.",{"type":126,"content":164},[165],"There is also a fourth case that is not a migration at all: the same model name changing behaviour. Chen, Zaharia and Zou compared the March and June 2023 versions of GPT-4 and found accuracy on identifying prime numbers dropping from 84% to 51%, along with more formatting mistakes in generated code. Their conclusion, that the behaviour of the \"same\" LLM service can change substantially in a short time, is the argument for running your regression suite on a schedule, not only when you switch.",{"type":133,"level":134,"id":95,"text":96},{"type":126,"content":168},[169],"How much warning you get depends on the provider and on the kind of model. From each provider's documentation:",{"type":171,"head":172,"rows":179},"table",[173,175,177],[174],"Provider",[176],"Minimum notice before retirement",[178],"How to monitor it",[180,195,201],[181,184,186],[182,183],"tag","children",[185],"At least 6 months for generally available models, 3 months for specialized variants (chat, Codex, deep research), as little as 2 weeks for preview models",[187,190,191,194],{"tag":159,"children":188},[189],"GET \u002Fv1\u002Fmodels"," returns an optional ",{"tag":159,"children":192},[193],"shutdown_date"," per model",[196,197,199],[182,183],[198],"At least 60 days for publicly released models; customers with active deployments are notified",[200],"Deprecations page with tentative retirement dates; Console usage export by API key and model",[202,203,205],[182,183],[204],"Preview models deprecated with at least 2 weeks' notice; the -latest alias is hot-swapped, with 2 weeks' email notice for breaking changes",[206],"Stable model strings \"usually don't change\"; deprecations page",{"type":126,"content":208},[209,210,213],"Two practical consequences. Never put a preview model or a ",{"tag":159,"children":211},[212],"-latest"," alias behind a business-critical feature: OpenAI's docs say outright that they do not recommend preview models for such workloads unless you can migrate on short notice. And put the retirement date of every model you use into a calendar the day you adopt it. Anthropic currently lists tentative dates as \"not sooner than\" a given day, which is a floor, not a promise.",{"type":133,"level":134,"id":98,"text":99},{"type":126,"content":216},[217],"A migration you cannot scope is a migration you cannot finish. Before any testing:",{"type":139,"ordered":219,"items":220},true,[221,226,242,247],[222,225],{"tag":144,"children":223},[224],"Pin exact model identifiers in production."," Stable names on Gemini, exact IDs on Claude and OpenAI. An alias moves without a deploy, which means behaviour can change without a commit.",[227,230,231,236,237,241],{"tag":144,"children":228},[229],"Centralise the model ID in configuration,"," one entry per use case (\"order-extraction\", \"support-triage\"), not a string literal per call site. A gateway such as ",{"tag":232,"to":233,"children":234},"link","\u002Ftools\u002Flitellm",[235],"LiteLLM"," or ",{"tag":232,"to":238,"children":239},"\u002Ftools\u002Fopenrouter",[240],"OpenRouter"," makes this a routing rule.",[243,246],{"tag":144,"children":244},[245],"Inventory actual usage."," Code search finds the call sites you know about. Anthropic's Console usage export (CSV by API key and model) and your gateway logs find the ones you forgot: the cron job, the internal tool, the prototype that became production.",[248,251],{"tag":144,"children":249},[250],"Record the request settings per use case:"," max tokens, thinking or effort settings, tool choice, structured output schema, sampling parameters. These are what the new model may reject.",{"type":133,"level":134,"id":101,"text":102},{"type":126,"content":254},[255],"I run every model swap through the same five stages. Each stage has an exit criterion, and any failure sends you back one step, not forward with a fix in production.",{"type":257,"attrs":258,"inner":262,"caption":263},"diagram",{"viewBox":259,"role":260,"aria-labelledby":261},"0 0 720 260","img","mig-flow-t mig-flow-d","\u003Ctitle id=\"mig-flow-t\">Five stages of an LLM model migration\u003C\u002Ftitle>\u003Cdesc id=\"mig-flow-d\">Inventory and pinning, then contract fixes until no request is rejected, then an offline regression suite comparing old and new model, then shadow traffic on live requests, then a canary rollout to a small share of users before full rollout. A failed gate sends the change back one stage; a feature flag allows rollback at the canary stage.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"ah-mig\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"10\" y=\"70\" width=\"132\" height=\"64\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"76\" y=\"98\" text-anchor=\"middle\" class=\"d-text\">1 inventory\u003C\u002Ftext>\u003Ctext x=\"76\" y=\"120\" text-anchor=\"middle\" class=\"d-small\">call sites, settings\u003C\u002Ftext>\u003Crect x=\"152\" y=\"70\" width=\"132\" height=\"64\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"218\" y=\"98\" text-anchor=\"middle\" class=\"d-text\">2 contract\u003C\u002Ftext>\u003Ctext x=\"218\" y=\"120\" text-anchor=\"middle\" class=\"d-small\">no 400s\u003C\u002Ftext>\u003Crect x=\"294\" y=\"70\" width=\"132\" height=\"64\" rx=\"10\" class=\"d-accent\" \u002F>\u003Ctext x=\"360\" y=\"98\" text-anchor=\"middle\" class=\"d-text\">3 offline evals\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"120\" text-anchor=\"middle\" class=\"d-small\">old vs new\u003C\u002Ftext>\u003Crect x=\"436\" y=\"70\" width=\"132\" height=\"64\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"502\" y=\"98\" text-anchor=\"middle\" class=\"d-text\">4 shadow\u003C\u002Ftext>\u003Ctext x=\"502\" y=\"120\" text-anchor=\"middle\" class=\"d-small\">live, not served\u003C\u002Ftext>\u003Crect x=\"578\" y=\"70\" width=\"132\" height=\"64\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"644\" y=\"98\" text-anchor=\"middle\" class=\"d-text\">5 canary\u003C\u002Ftext>\u003Ctext x=\"644\" y=\"120\" text-anchor=\"middle\" class=\"d-small\">5% to 100%\u003C\u002Ftext>\u003Cpath d=\"M142 102H160\" class=\"d-line\" marker-end=\"url(#ah-mig)\" \u002F>\u003Cpath d=\"M284 102H302\" class=\"d-line\" marker-end=\"url(#ah-mig)\" \u002F>\u003Cpath d=\"M426 102H444\" class=\"d-line\" marker-end=\"url(#ah-mig)\" \u002F>\u003Cpath d=\"M568 102H586\" class=\"d-line\" marker-end=\"url(#ah-mig)\" \u002F>\u003Cpath d=\"M502 136V170H218V138\" class=\"d-line d-dash\" marker-end=\"url(#ah-mig)\" \u002F>\u003Ctext x=\"360\" y=\"190\" text-anchor=\"middle\" class=\"d-label\">gate failed: back one stage\u003C\u002Ftext>\u003Cpath d=\"M644 136V220H76V138\" class=\"d-line-accent d-dash\" marker-end=\"url(#ah-mig)\" \u002F>\u003Ctext x=\"360\" y=\"240\" text-anchor=\"middle\" class=\"d-label\">flag: instant rollback\u003C\u002Ftext>",[264],"A model swap as a release pipeline. Stages 1 and 2 are mechanical; stages 3 to 5 are where behaviour changes are caught.",{"type":133,"level":134,"id":104,"text":105},{"type":126,"content":267},[268],"Contract changes are deterministic, so handle them first and separately. Read the provider's migration guide for your exact source and target model. As a concrete example, these are some of the changes the Claude Opus 5.5 guide lists for code coming from earlier models:",{"type":171,"head":270,"rows":277},[271,273,275],[272],"Old setting",[274],"On Claude Opus 5.5",[276],"Fix",[278,292,305,321,327],[279,288,290],[280,283,284,287],{"tag":159,"children":281},[282],"thinking: {type: \"disabled\"}"," or a ",{"tag":159,"children":285},[286],"budget_tokens"," budget",[289],"400 error",[291],"Remove the field; choose an effort level (default is medium)",[293,298,299],[294,297],{"tag":159,"children":295},[296],"tool_choice"," any or a named tool",[289],[300,301,304],"Use ",{"tag":159,"children":302},[303],"auto"," plus instructions, or structured outputs",[306,318,319],[307,310,311,310,314,317],{"tag":159,"children":308},[309],"temperature",", ",{"tag":159,"children":312},[313],"top_p",{"tag":159,"children":315},[316],"top_k"," at non-default values",[289],[320],"Omit them; steer with the prompt",[322,324,325],[323],"Assistant prefill to force a format",[289],[326],"Structured outputs or system prompt instructions",[328,332,334],[329,331],{"tag":159,"children":330},[161]," sized for an older tokenizer",[333],"Up to about 35% more tokens for the same text versus pre-Opus 4.7 models",[335],"Re-measure with token counting; add headroom",{"type":126,"content":337},[338,339,342],"The exit criterion for this stage is simple: every recorded request shape from the inventory goes through against the new model without an error. Anthropic also ships a ",{"tag":159,"children":340},[341],"\u002Fclaude-api migrate"," skill for Claude Code that applies the model ID swap and breaking parameter changes across a codebase and produces a checklist of items to verify by hand. Tools like that speed up stage 2; they do not replace stage 3.",{"type":133,"level":134,"id":107,"text":108},{"type":126,"content":345},[346],"Anthropic's eval guide describes regression evals as the ones that should sit near 100% and catch backsliding. For a model swap, that suite runs both models on the same inputs and compares. Build it from production, not from imagination: sample real requests per use case, add every past incident, and keep the expected outcome or a reference answer next to each one. Then measure more than accuracy:",{"type":171,"head":348,"rows":355},[349,351,353],[350],"Metric",[352],"Why it moves on a model swap",[354],"Example gate (my defaults)",[356,363,370,377,384,391],[357,359,361],[358],"Task success on the golden set",[360],"Behaviour change",[362],"No drop beyond noise; every past incident still passes",[364,366,368],[365],"Schema and parse failures",[367],"Formatting habits differ between models",[369],"Zero, with structured outputs on",[371,373,375],[372],"Refusal and abstention rate",[374],"Models differ in caution",[376],"Within a few points of the old model; read every new refusal",[378,380,382],[379],"Output length",[381],"Verbosity changes break UIs and SMS limits",[383],"Median within ±20%",[385,387,389],[386],"Tokens and cost per completed task",[388],"Tokenizer, verbosity, thinking",[390],"Known and accepted before rollout",[392,394,396],[393],"Latency p50 and p95",[395],"Thinking defaults, model size",[397],"Within the feature's latency budget",{"type":126,"content":399},[400],"Score open-ended outputs pairwise: show a judge the old and the new answer for the same input, in random order, and ask which better meets the rubric. Pairwise comparison is more stable than absolute scores for \"is this as good as before\", and it surfaces the cases worth reading. The harness itself is small:",{"type":159,"code":402},"import random, statistics\n\ndef compare(cases, old_model, new_model, run, judge, checks):\n    \"\"\"run(model, case) -> Result(text, tokens, ms, refused); judge(case, a, b) -> 'a'|'b'|'tie'\"\"\"\n    rows = []\n    for case in cases:\n        old, new = run(old_model, case), run(new_model, case)\n        flip = random.random() \u003C 0.5                      # hide which answer is new\n        verdict = judge(case, *( (new.text, old.text) if flip else (old.text, new.text) ))\n        new_wins = (verdict == \"a\") == flip if verdict != \"tie\" else None\n        rows.append({\n            \"id\": case.id,\n            \"checks_failed\": [c.name for c in checks if not c(case, new)],   # schema, refs, rules\n            \"new_wins\": new_wins,\n            \"refused\": (old.refused, new.refused),\n            \"len_ratio\": len(new.text) \u002F max(1, len(old.text)),\n            \"token_ratio\": new.tokens \u002F max(1, old.tokens),\n            \"ms\": (old.ms, new.ms),\n        })\n    losses = [r for r in rows if r[\"new_wins\"] is False or r[\"checks_failed\"]]\n    print(\"losses to read:\", [r[\"id\"] for r in losses])\n    print(\"median length ratio:\", statistics.median(r[\"len_ratio\"] for r in rows))\n    return rows",{"type":126,"content":404},[405,406,410,411,415],"Run each case more than once if the feature is agentic or sampled; one run per model hides variance. And keep the losses list as the real output of the stage: the decision to proceed is made by reading those cases, not by a single aggregate score. ",{"tag":232,"to":407,"children":408},"\u002Ftools\u002Fpromptfoo",[409],"Promptfoo"," and similar tools can run the same side-by-side comparison from configuration if you prefer not to maintain a harness; the method is covered in more depth in ",{"tag":232,"to":412,"children":413},"\u002Fblog\u002Fllm-evals-for-product-features",[414],"LLM evals for product features",".",{"type":133,"level":134,"id":110,"text":111},{"type":126,"content":418},[419],"When the suite shows losses, resist adding instructions on top of the old prompt. Prompts accumulate workarounds for the previous model's quirks (\"never use bullet points\", \"ALWAYS return the field even if empty\"), and a newer model may follow them too literally. That fits the Sclar et al. finding that prompt-format preferences transfer poorly between models. My order of work:",{"type":139,"ordered":219,"items":421},[422,427,432,437],[423,426],{"tag":144,"children":424},[425],"Remove workarounds first"," and re-run. Many losses disappear.",[428,431],{"tag":144,"children":429},[430],"Re-tune the reasoning setting."," On Claude, effort replaces thinking budgets, and defaults differ between models (medium on Opus 5.5). Sweep low, medium and high on the suite and pick per use case.",[433,436],{"tag":144,"children":434},[435],"Replace format coaxing with structured outputs"," where the provider supports them.",[438,441],{"tag":144,"children":439},[440],"Only then add new instructions,"," one at a time, each tied to a failing case.",{"type":443,"variant":444,"title":445,"body":446},"callout","warn","Version the prompt with the model",[447],[448],"A prompt is tuned for a model, not in general. Store prompt version and model ID together, deploy them together and roll them back together. Rolling back the model while keeping the new prompt is a third configuration nobody tested.",{"type":133,"level":134,"id":113,"text":114},{"type":126,"content":451},[452,453,456],"Offline suites only contain the cases you thought of. ",{"tag":144,"children":454},[455],"Shadow traffic"," sends a copy of live requests to the new model, logs its answer next to the served one, and never shows it to users. Run it for a few days across the weekly traffic pattern, then run the same checks as in stage 3 on the shadow pairs. Two cautions: shadow calls cost money, so sample if volume is high; and never let the shadow path execute side effects. For agents, shadow only the read-only part or point tools at a sandbox.",{"type":126,"content":458},[459,460,463,464,468],"The ",{"tag":144,"children":461},[462],"canary"," then serves the new model to a small share of traffic behind a feature flag, with the same dashboards you use for the old model: error rate, refusal rate, user feedback, cost per task and latency. Widen in steps (5%, 25%, 100%) with a fixed observation window per step, and keep the old configuration deployable until the retirement date, so rollback is a flag flip rather than a release. ",{"tag":232,"to":465,"children":466},"\u002Fblog\u002Fagent-observability-opentelemetry",[467],"Tracing with OpenTelemetry"," makes the side-by-side comparison per model a query instead of a project.",{"type":133,"level":134,"id":116,"text":117},{"type":126,"content":471},[472,473,477],"The regression suite is the expensive part to build and the cheap part to keep. Run it weekly against the production configuration, and on every prompt or model change. That catches drift behind a stable name (the Chen et al. case), it gives you a baseline when the next deprecation notice arrives, and it turns \"should we switch to the new, cheaper model?\" into a measurement. The same suite is what makes ",{"tag":232,"to":474,"children":475},"\u002Fblog\u002Fllm-cost-latency-prompt-caching-routing",[476],"model routing for cost"," safe: a cheaper model is only cheaper if it passes.",{"type":126,"content":479},[480,481,485],"Leaderboards help you pick candidates, as I discussed for ",{"tag":232,"to":482,"children":483},"\u002Fblog\u002Fartificial-analysis-leaderboard-claude-opus-5-5",[484],"the current Artificial Analysis ranking",", but they cannot tell you whether your extraction prompt still finds the delivery date. Only your suite can.",{"type":133,"level":134,"id":119,"text":120},{"type":139,"ordered":219,"items":488},[489,491,493,495,497,499,501,503,505],[490],"Calendar the retirement date of every model you use; avoid preview models and -latest aliases in production.",[492],"Pin exact model IDs, centralise them per use case, and inventory real usage from provider exports and gateway logs.",[494],"Read the migration guide for your exact source and target model; fix every rejected parameter first.",[496],"Re-measure token counts and adjust max_tokens, budgets and compaction triggers.",[498],"Run old and new model side by side on a golden set built from production and past incidents.",[500],"Gate on success, parse failures, refusals, length, cost per task and latency, and read every loss.",[502],"Remove old prompt workarounds before adding new instructions; re-tune effort per use case.",[504],"Shadow live traffic without side effects, then canary behind a flag with rollback until the retirement date.",[506],"Keep the suite running weekly after the migration.",{"type":126,"content":508},[509,510,415],"If you have a retirement date coming up and want a second pair of hands on the migration, see ",{"tag":232,"to":511,"children":512},"\u002Fexpertise\u002Fai-engineer",[513],"AI engineering",{"type":133,"level":134,"id":122,"text":123},{"type":139,"ordered":219,"items":516},[517,521,524,527,530,533,536],[518],{"tag":519,"href":38,"children":520},"a",[37],[522],{"tag":519,"href":41,"children":523},[40],[525],{"tag":519,"href":44,"children":526},[43],[528],{"tag":519,"href":47,"children":529},[46],[531],{"tag":519,"href":50,"children":532},[49],[534],{"tag":519,"href":53,"children":535},[52],[537],{"tag":519,"href":56,"children":538},[55],[540,593,648,695],{"slug":541,"published":5,"minutes":542,"category":7,"tags":543,"keywords":549,"about":561,"sources":568,"cover":587,"og":588,"expertise":59,"locales":589,"lang":61,"title":590,"description":591,"coverAlt":592},"structured-outputs-json-schema-llm",9,[544,545,546,547,548],"Structured outputs","JSON Schema","Constrained decoding","LLM APIs","Validation",[550,551,552,553,554,555,556,557,558,559,560],"structured outputs LLM","JSON schema LLM output","OpenAI structured outputs strict mode","Claude structured outputs","Gemini structured output","constrained decoding","JSON mode vs structured outputs","validate LLM JSON output","Zod structured output","Pydantic LLM output","vLLM structured outputs",[562,565,567],{"name":563,"url":564},"JSON","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FJSON",{"name":545,"url":566},"https:\u002F\u002Fjson-schema.org\u002F",{"name":30,"url":31},[569,572,575,578,581,584],{"title":570,"url":571},"OpenAI API docs: Structured model outputs","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fstructured-outputs",{"title":573,"url":574},"Claude API docs: Structured outputs","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fstructured-outputs",{"title":576,"url":577},"Gemini API docs: Structured outputs","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fstructured-output",{"title":579,"url":580},"vLLM docs: Structured Outputs","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fstructured_outputs.html",{"title":582,"url":583},"Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.02442",{"title":585,"url":586},"Geng et al.: JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.10868","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fog.jpg",[61,62,63],"Structured outputs with JSON Schema: what the guarantee covers, and what it does not","How structured outputs and constrained decoding work on OpenAI, Claude, Gemini and vLLM, which schema rules matter, and what you still validate.","A JSON Schema compiled into a grammar that masks tokens, producing schema-valid JSON that then passes a business validation step.",{"slug":594,"published":5,"minutes":542,"category":7,"tags":595,"keywords":601,"about":614,"sources":622,"cover":642,"og":643,"expertise":59,"locales":644,"lang":61,"title":645,"description":646,"coverAlt":647},"evaluating-ai-agents-trajectories",[596,597,598,599,600],"Agent evals","Trajectories","Tool calling","pass^k","LLM-as-judge",[602,603,604,605,606,607,608,609,610,611,612,613],"evaluating AI agents","agent trajectory evaluation","agent evals","tool call accuracy","trajectory match evaluator","pass^k metric","tau-bench","LLM as a judge for agents","agent regression testing","outcome vs trajectory evaluation","agentevals","tool correctness metric",[615,618,621],{"name":616,"url":617},"Intelligent agent","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIntelligent_agent",{"name":619,"url":620},"Software testing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSoftware_testing",{"name":30,"url":31},[623,624,627,630,633,636,639],{"title":55,"url":56},{"title":625,"url":626},"Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045",{"title":628,"url":629},"Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07982",{"title":631,"url":632},"LangChain: agentevals (GitHub)","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals",{"title":634,"url":635},"DeepEval docs: Tool Correctness","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fmetrics-tool-correctness",{"title":637,"url":638},"Ragas docs: Agentic or tool use metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002Fagents\u002F",{"title":640,"url":641},"OpenAI API docs: Trace grading","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ftrace-grading","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fog.jpg",[61,62,63],"Evaluating AI agents: grading trajectories, outcomes and reliability","How to evaluate AI agents: outcome vs trajectory graders, trajectory match modes, pass^k, isolated environments and LLM judges that hold up.","An agent trajectory of tool calls graded twice: a trajectory grader checks order and arguments, an outcome grader compares the database with the goal state.",{"slug":649,"published":5,"minutes":650,"category":7,"tags":651,"keywords":657,"about":668,"sources":676,"cover":689,"og":690,"expertise":59,"locales":691,"lang":61,"title":692,"description":693,"coverAlt":694},"semantic-caching-llm-responses",6,[652,653,654,655,656],"Semantic cache","LLM cost","Latency","Embeddings","Redis",[658,659,660,661,662,663,664,665,666,667],"semantic caching llm","semantic cache","llm response cache","gptcache","semantic cache similarity threshold","prompt caching vs semantic caching","reduce llm api cost","cache llm answers redis","semantic cache false positive","llm cache invalidation",[669,672,675],{"name":670,"url":671},"Cache (computing)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCache_(computing)",{"name":673,"url":674},"Sentence embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSentence_embedding",{"name":30,"url":31},[677,680,683,686],{"title":678,"url":679},"Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2411.05276",{"title":681,"url":682},"Zilliz: GPTCache README (modules, similarity evaluation, eviction)","https:\u002F\u002Fgithub.com\u002Fzilliztech\u002FGPTCache",{"title":684,"url":685},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching",{"title":687,"url":688},"pgvector README: vector types and indexes","https:\u002F\u002Fgithub.com\u002Fpgvector\u002Fpgvector","\u002Fimages\u002Fblog\u002Fsemantic-caching-llm-responses\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fsemantic-caching-llm-responses\u002Fog.jpg",[61,62,63],"Semantic caching for LLM apps: when reusing answers is safe, and when it is not","How semantic caches reuse LLM answers for similar questions, how they differ from prompt caching, and how to avoid wrong hits, leaks and stale answers.","A new question is embedded and compared with cached questions; above a similarity threshold the stored answer is returned, below it the model is called.",{"slug":696,"published":697,"minutes":6,"category":7,"tags":698,"keywords":704,"about":713,"sources":723,"cover":793,"og":794,"expertise":59,"locales":795,"lang":61,"title":796,"description":797,"coverAlt":798},"self-hosted-llm-gdpr-cost","2026-10-09",[699,700,701,702,703],"Self-hosting","GDPR","vLLM","GPU cost","Open-weight models",[705,706,707,708,709,710,711,712],"self-hosted LLM GDPR","self-host LLM data protection","LLM GPU memory requirements","vLLM throughput benchmark","EU GPU cloud for LLMs","LLM API vs self-hosting break-even","open-weight LLM hardware cost","data processing agreement LLM",[714,717,718,721],{"name":715,"url":716},"General Data Protection Regulation","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGeneral_Data_Protection_Regulation",{"name":30,"url":31},{"name":719,"url":720},"Graphics processing unit","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGraphics_processing_unit",{"name":701,"url":722},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm",[724,727,730,733,736,739,742,745,748,751,754,757,760,763,766,769,772,775,778,781,784,787,790],{"title":725,"url":726},"openai\u002Fgpt-oss-120b model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fopenai\u002Fgpt-oss-120b",{"title":728,"url":729},"mistralai\u002FMistral-Small-3.2-24B-Instruct-2506 model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-Small-3.2-24B-Instruct-2506",{"title":731,"url":732},"meta-llama\u002FLlama-3.3-70B-Instruct model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FLlama-3.3-70B-Instruct",{"title":734,"url":735},"mistralai\u002FMistral-Large-3-675B-Instruct-2512 model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-Large-3-675B-Instruct-2512",{"title":737,"url":738},"vLLM blog: v0.6.0 performance update (September 2024)","https:\u002F\u002Fblog.vllm.ai\u002F2024\u002F09\u002F05\u002Fperf-update.html",{"title":740,"url":741},"vLLM documentation: engine arguments","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fconfiguration\u002Fengine_args.html",{"title":743,"url":744},"vLLM documentation: vllm bench serve","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fbench\u002Fserve.html",{"title":746,"url":747},"SemiAnalysis InferenceX: Llama 3.3 70B on H100 vs H200","https:\u002F\u002Finferencex.semianalysis.com\u002Fcompare\u002Fllama-3-3-70b-h100-vs-h200",{"title":749,"url":750},"Scaleway GPU instances pricing","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fpricing\u002Fgpu\u002F",{"title":752,"url":753},"Scaleway blog: a transparent update on Scaleway pricing (27 April 2026)","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fblog\u002Fa-transparent-update-on-scaleway-pricing\u002F",{"title":755,"url":756},"Scaleway Generative APIs pricing","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fpricing\u002Fmodel-as-a-service\u002F",{"title":758,"url":759},"OVHcloud public cloud price list","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fprices\u002F",{"title":761,"url":762},"OVHcloud AI Endpoints","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fai-endpoints\u002F",{"title":764,"url":765},"OVHcloud AI Endpoints model catalogue","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fai-endpoints\u002Fcatalog\u002F",{"title":767,"url":768},"Hetzner dedicated GPU servers","https:\u002F\u002Fwww.hetzner.com\u002Fdedicated-rootserver\u002Fmatrix-gpu\u002F",{"title":770,"url":771},"NVIDIA newsroom: GeForce RTX 50 Series launch","https:\u002F\u002Fnvidianews.nvidia.com\u002Fnews\u002Fnvidia-blackwell-geforce-rtx-50-series-opens-new-world-of-ai-computer-graphics",{"title":773,"url":774},"NVIDIA GeForce RTX 5090","https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fgeforce\u002Fgraphics-cards\u002F50-series\u002Frtx-5090\u002F",{"title":776,"url":777},"NVIDIA GeForce software licence","https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fdrivers\u002Fgeforce-license\u002F",{"title":779,"url":780},"GDPR Article 28: processor","https:\u002F\u002Fgdpr-info.eu\u002Fart-28-gdpr\u002F",{"title":782,"url":783},"GDPR Article 44: general principle for transfers","https:\u002F\u002Fgdpr-info.eu\u002Fart-44-gdpr\u002F",{"title":785,"url":786},"European Commission: EU-US data transfers (Data Privacy Framework)","https:\u002F\u002Fcommission.europa.eu\u002Flaw\u002Flaw-topic\u002Fdata-protection\u002Finternational-dimension-data-protection\u002Feu-us-data-transfers_en",{"title":788,"url":789},"EDPB Opinion 28\u002F2024 on AI models (adopted 17 December 2024)","https:\u002F\u002Fwww.edpb.europa.eu\u002Fsystem\u002Ffiles\u002F2024-12\u002Fedpb_opinion_202428_ai-models_en.pdf",{"title":791,"url":792},"Mistral AI pricing","https:\u002F\u002Fmistral.ai\u002Fpricing","\u002Fimages\u002Fblog\u002Fself-hosted-llm-gdpr-cost\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fself-hosted-llm-gdpr-cost\u002Fog.jpg",[61,62,63],"Self-hosting LLMs for GDPR: when it is required and what it costs","Self-hosting an LLM for GDPR: when it is required, GPU memory for open-weight models, EU prices as of October 2026 and break-even per million tokens.","Cover art for self-hosted LLMs under GDPR: a decision between your own GPUs and an EU API, with a break-even line.",1792177152115]