[{"data":1,"prerenderedAt":652},["ShallowReactive",2],{"blog-prompt-versioning-management-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":25,"sources":35,"cover":60,"og":61,"expertise":62,"locales":63,"lang":64,"title":67,"description":68,"coverAlt":69,"metaTitle":70,"takeaways":71,"faq":77,"toc":93,"blocks":124,"others":417},"prompt-versioning-management","2026-10-17",8,"llmops",[9,10,11,12,13],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[15,16,17,18,19,20,21,22,23,24],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[26,29,32],{"name":27,"url":28},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":30,"url":31},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":33,"url":34},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[36,39,42,45,48,51,54,57],{"title":37,"url":38},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":40,"url":41},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":43,"url":44},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":46,"url":47},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":49,"url":50},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":52,"url":53},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":55,"url":56},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":58,"url":59},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg","ai-engineer",[64,65,66],"en","de","hu","Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.","Prompt versioning: git, registries, labels · Balázs Csorba",[72,73,74,75,76],"A prompt version is the whole bundle: template, model, parameters, tools and output schema. Version them together or you cannot reproduce a result.","OpenAI is retiring reusable prompt objects, with v1\u002Fprompts scheduled to shut down on 30 November 2026, and now recommends keeping prompts in application code.","Registries such as Langfuse and MLflow add immutable versions and movable labels, which make rollbacks a label change instead of a deploy.","Gate every prompt change with an eval run on the pull request, and link production traces to the prompt version that produced them.","Changing a system prompt or tool definition invalidates provider prompt caches, so a prompt release is also a short cost and latency event.",[78,81,84,87,90],{"q":79,"a":80},"Should prompts live in git or in a prompt management tool?","Put them in git by default: review, history and tests come for free, and OpenAI now recommends exactly that. A registry is worth it when non-developers edit prompts, when you need to roll back without a deploy, or when you want production metrics per version. Many teams do both: git as the source, a registry as the runtime store.",{"q":82,"a":83},"What is the difference between a prompt version and a label?","A version is an immutable snapshot of the prompt; MLflow states that versions cannot be edited after creation. A label or alias such as production is a movable pointer to one version. Your code asks for the label, and promoting or rolling back means moving the pointer.",{"q":85,"a":86},"What happens to OpenAI saved prompts?","OpenAI is deprecating reusable prompt objects in the API. Prompt creation is de-emphasized from 3 June 2026 and the v1\u002Fprompts endpoint is scheduled to shut down on 30 November 2026. The migration guide says to move prompt text into your code, replace variables with typed function arguments and pass the result as input to the Responses API.",{"q":88,"a":89},"Does fetching prompts from a registry add latency?","Not much if the SDK caches. Langfuse caches prompts client-side with a default TTL of 60 seconds and refreshes them in the background, serving the stale version meanwhile. If the cache is empty and the API is down, a fallback prompt you pass in keeps the application running.",{"q":91,"a":92},"How do I test a prompt change before it reaches users?","Run the same evaluation set against the old and new prompt on every pull request that touches prompt files. The promptfoo GitHub Action does this and posts the comparison as a PR comment. After release, compare scores, latency and cost per prompt version on linked traces.",[94,97,100,103,106,109,112,115,118,121],{"id":95,"title":96},"what-is-a-version","What belongs in a prompt version",{"id":98,"title":99},"git-or-registry","Git or a prompt registry?",{"id":101,"title":102},"versions-and-labels","Immutable versions, movable labels",{"id":104,"title":105},"runtime-fetch","Fetching prompts at runtime without new failure modes",{"id":107,"title":108},"prompts-in-code","If prompts live in code, structure them",{"id":110,"title":111},"eval-gate","Gate every prompt change with an eval",{"id":113,"title":114},"link-to-traces","Link production traces to prompt versions",{"id":116,"title":117},"cache-side-effects","A prompt release is also a cache event",{"id":119,"title":120},"my-default","A sensible default for small teams",{"id":122,"title":123},"sources","Sources",[125,129,132,135,138,176,179,180,188,196,240,241,255,264,276,277,280,282,285,286,293,295,296,304,306,319,320,323,331,332,350,357,358,364,388,389],{"type":126,"content":127},"paragraph",[128],"Most LLM features change more often in their prompts than in their code. A support assistant gets a new escalation rule, an extraction step learns about a new invoice layout, a classifier gets one more category. Each of these is a behaviour change shipped to production, and in many codebases it is the least controlled one: a string edited in a dashboard, a template pasted into a config file, no review and no record of which version answered a given customer last Tuesday.",{"type":126,"content":130},[131],"This guide covers what a prompt version should contain, when git is enough and when a prompt registry pays for itself, how labels make rollbacks cheap, how to gate prompt changes with evals in CI, and one side effect teams often miss: a prompt release resets your provider-side prompt cache. It also covers a 2026 change that settles part of the debate, OpenAI retiring its saved prompt objects.",{"type":133,"level":134,"id":95,"text":96},"heading",2,{"type":126,"content":136},[137],"A template string alone does not determine behaviour. The same text produces different output with a different model, temperature, tool list or output schema. If your version history records only the text, you cannot answer the question that matters during an incident: what exactly ran? Version the bundle:",{"type":139,"head":140,"rows":145},"table",[141,143],[142],"Part",[144],"Why it belongs in the version",[146,151,156,161,166,171],[147,149],[148],"System and user templates",[150],"The instructions themselves",[152,154],[153],"Model id",[155],"A prompt tuned for one model is a different prompt on another",[157,159],[158],"Parameters",[160],"Temperature, max tokens, reasoning effort change output",[162,164],[163],"Tool definitions",[165],"Descriptions steer tool choice as much as the system prompt",[167,169],[168],"Output schema",[170],"Downstream code depends on it",[172,174],[173],"Few-shot examples",[175],"Often the largest and most edited part",{"type":126,"content":177},[178],"Both registries discussed below support this. Langfuse prompts carry a config field next to the template, and MLflow lets each version carry a commit message much like a git commit. In git you get it by keeping the template, parameters and schema in the same module.",{"type":133,"level":134,"id":98,"text":99},{"type":126,"content":181},[182,183,187],"In 2026 the strongest argument for git came from a vendor that used to sell the alternative. OpenAI is deprecating reusable prompt objects in its API: prompt creation is de-emphasized from 3 June 2026 and ",{"tag":184,"children":185},"code",[186],"v1\u002Fprompts"," is scheduled to shut down on 30 November 2026. Its migration guide tells teams to move prompt content into application code so prompt changes go through the same review and release process as other code, to replace prompt variables with typed function arguments, and to manage versions with commits, pull requests and evals in CI.",{"type":126,"content":189},[190,191,195],"Registries still solve real problems. ",{"tag":192,"to":193,"children":194},"link","\u002Ftools\u002Flangfuse",[13]," and MLflow store immutable versions with movable labels, so a product manager can test a wording change and an on-call engineer can roll back in seconds without a deploy. They also join production metrics to versions. The question is not which camp is right, but who edits prompts and how fast a rollback must be.",{"type":139,"head":197,"rows":204},[198,200,202],[199],"Criterion",[201],"Prompts in git",[203],"Prompt registry",[205,212,219,226,233],[206,208,210],[207],"Who edits",[209],"Developers",[211],"Developers and domain experts",[213,215,217],[214],"Review",[216],"Pull request, same as code",[218],"Tool-specific, often weaker",[220,222,224],[221],"Rollback",[223],"Revert and deploy",[225],"Move a label, effective after cache TTL",[227,229,231],[228],"Runtime dependency",[230],"None",[232],"Registry API, mitigated by cache and fallback",[234,236,238],[235],"Metrics per version",[237],"Only if you tag traces yourself",[239],"Built in when traces are linked",{"type":133,"level":134,"id":101,"text":102},{"type":126,"content":242},[243,244,247,248,251,252,254],"The useful mental model is the same one container registries use: versions are immutable, tags move. MLflow states that prompt versions are immutable for reproducibility and that an alias such as ",{"tag":184,"children":245},[246],"production"," is a mutable reference you can repoint for A\u002FB tests or rollbacks. Langfuse assigns version numbers automatically, maintains a ",{"tag":184,"children":249},[250],"latest"," label pointing at the newest version, and returns the version labelled ",{"tag":184,"children":253},[246]," when you fetch without a label or version.",{"type":256,"attrs":257,"inner":261,"caption":262},"diagram",{"viewBox":258,"role":259,"aria-labelledby":260},"0 0 720 170","img","pvm1-t pvm1-d","\u003Ctitle id=\"pvm1-t\">From commit to production label\u003C\u002Ftitle>\u003Cdesc id=\"pvm1-d\">A prompt change is committed, evaluated on the pull request, registered as a new immutable version, labelled staging, and promoted by moving the production label.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"pvm1-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"20\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"79\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Commit\u003C\u002Ftext>\u003Ctext x=\"79\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">prompt + config\u003C\u002Ftext>\u003Cpath d=\"M138 85H158\" class=\"d-line\" marker-end=\"url(#pvm1-ah)\" \u002F>\u003Crect x=\"160\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"220\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Eval on PR\u003C\u002Ftext>\u003Ctext x=\"220\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">old vs new\u003C\u002Ftext>\u003Cpath d=\"M279 85H299\" class=\"d-line\" marker-end=\"url(#pvm1-ah)\" \u002F>\u003Crect x=\"301\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"360\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Version 14\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">immutable\u003C\u002Ftext>\u003Cpath d=\"M419 85H439\" class=\"d-line\" marker-end=\"url(#pvm1-ah)\" \u002F>\u003Crect x=\"441\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"500\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">staging\u003C\u002Ftext>\u003Ctext x=\"500\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">label\u003C\u002Ftext>\u003Cpath d=\"M560 85H580\" class=\"d-line\" marker-end=\"url(#pvm1-ah)\" \u002F>\u003Crect x=\"582\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box d-fill-accent\" \u002F>\u003Ctext x=\"641\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">production\u003C\u002Ftext>\u003Ctext x=\"641\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">move label\u003C\u002Ftext>",[263],"Rolling back means pointing the production label at version 13 again; nothing is rebuilt.",{"type":126,"content":265},[266,267,269,270,272,273,275],"Two details from the Langfuse documentation are worth designing around. Requesting a label that no version carries returns a 404; there is no silent fallback to ",{"tag":184,"children":268},[246]," or ",{"tag":184,"children":271},[250],". And protected labels, available in its paid and enterprise editions, stop viewer and member roles from moving ",{"tag":184,"children":274},[246],", which is the registry equivalent of a protected branch.",{"type":133,"level":134,"id":104,"text":105},{"type":126,"content":278},[279],"A registry puts a network call between your feature and its instructions. Langfuse handles that with a client-side cache: the default TTL is 60 seconds, and when it expires the SDK keeps serving the stale prompt while it refetches in the background. If the cache is empty and the API is unreachable, a fallback prompt you pass in is returned instead. That last case is the one to plan for, because it happens on cold start in exactly the incident where the registry is down.",{"type":184,"code":281},"from langfuse import get_client\nfrom prompts.support import SUPPORT_SYSTEM_FALLBACK  # the same text, committed in git\n\nlangfuse = get_client()\n\ndef support_prompt():\n    # label, not version: promotion and rollback happen by moving \"production\"\n    return langfuse.get_prompt(\n        \"support-assistant\",\n        label=\"production\",\n        cache_ttl_seconds=300,          # default is 60; longer means slower rollbacks\n        fallback=SUPPORT_SYSTEM_FALLBACK,\n    )",{"type":126,"content":283},[284],"The TTL is a trade-off you should write down: the longer it is, the fewer registry calls you make and the longer a rollback takes to reach every instance. Pin an explicit version in batch jobs and evals, where reproducibility matters more than following the label.",{"type":133,"level":134,"id":107,"text":108},{"type":126,"content":287},[288,289,292],"OpenAI's guide is specific: keep prompts in a small ",{"tag":184,"children":290},[291],"prompts\u002F"," module with one named builder per prompt and lightweight eval fixtures, and do not scatter them inline across the codebase. Add two habits. First, put static content first and dynamic content last, which the same guide recommends because cache hits depend on exact prefix matches. Second, export a version identifier with each prompt, so traces can record which one ran even without a registry.",{"type":184,"code":294},"\u002F\u002F prompts\u002Finvoice-extract.ts\nimport { z } from 'zod'\n\nexport const INVOICE_EXTRACT = {\n  id: 'invoice-extract',\n  version: '2026-10-14.2',          \u002F\u002F bump on any change to text, model, params or schema\n  model: 'claude-sonnet-5-5',\n  params: { max_tokens: 2048 },\n  schema: z.object({ supplier: z.string(), total: z.number(), currency: z.string().length(3) }),\n  build: (input: { pdfText: string; locale: 'de' | 'en' | 'hu' }) => ({\n    system: SYSTEM_RULES,           \u002F\u002F static, long, cacheable: first\n    messages: [{ role: 'user' as const, content: `Locale: ${input.locale}\\n\\n${input.pdfText}` }],\n  }),\n}",{"type":133,"level":134,"id":110,"text":111},{"type":126,"content":297},[298,299,303],"Versioning tells you what changed; only an evaluation tells you whether it got better. The cheapest workable setup runs a fixed test set against the old and new prompt on every pull request that touches prompt files. The ",{"tag":192,"to":300,"children":301},"\u002Ftools\u002Fpromptfoo",[302],"Promptfoo"," GitHub Action does exactly this: it triggers on changes under your prompt paths, compares prompts before and after the change, posts the result as a PR comment with a link to its viewer, and caches model calls between runs to keep cost down.",{"type":184,"code":305},"# .github\u002Fworkflows\u002Fprompt-eval.yml (trimmed)\non:\n  pull_request:\n    paths: ['prompts\u002F**']\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    permissions: { pull-requests: write }\n    steps:\n      - uses: actions\u002Fcheckout@v4\n      - uses: promptfoo\u002Fpromptfoo-action@v1\n        with:\n          github-token: ${{ secrets.GITHUB_TOKEN }}\n          prompts: 'prompts\u002F**\u002F*.json'\n          config: 'prompts\u002Fpromptfooconfig.yaml'\n          cache-path: ~\u002F.cache\u002Fpromptfoo",{"type":126,"content":307},[308,309,313,314,318],"What goes into the test set matters more than the tool. Start from real failures, as described in ",{"tag":192,"to":310,"children":311},"\u002Fblog\u002Fllm-evals-for-product-features",[312],"LLM evals for product features",": every bug report that a prompt change fixed becomes a case, so the next change cannot quietly bring it back. A model upgrade is just a prompt version with a new model id, and gets the same gate, see ",{"tag":192,"to":315,"children":316},"\u002Fblog\u002Fllm-model-migration-regression-testing",[317],"model migrations without regressions",".",{"type":133,"level":134,"id":113,"text":114},{"type":126,"content":321},[322],"Offline evals cover the cases you thought of. Production covers the rest, but only if each generation records which prompt version produced it. Langfuse links a prompt to a generation through the SDK or its OpenAI, LangChain and AI SDK integrations, and its metrics tab then compares versions on median latency, input and output tokens, cost, generation count and median score. One caveat from the documentation: when the fallback prompt is used, no link is created, so a spike of unlinked generations is itself a signal that the registry was unreachable.",{"type":126,"content":324},[325,326,330],"Without a registry, put the prompt id and version from your code into the span attributes. The OpenTelemetry setup in ",{"tag":192,"to":327,"children":328},"\u002Fblog\u002Fagent-observability-opentelemetry",[329],"observability for LLM agents"," gives you the same comparison with a group-by.",{"type":133,"level":134,"id":116,"text":117},{"type":126,"content":333},[334,335,338,339,338,342,345,346,318],"Provider prompt caches key on exact prefixes. Anthropic caches ",{"tag":184,"children":336},[337],"tools",", then ",{"tag":184,"children":340},[341],"system",{"tag":184,"children":343},[344],"messages",", and a change at one level invalidates that level and everything after it: edit a tool description and the whole cache is gone, edit the system prompt and system plus messages are rebuilt. Cache writes cost more than normal input (1.25 times base for the five-minute cache) and reads cost a fraction, so right after a release you pay write prices and see longer time to first token until the new prefix is warm. The mechanics are covered in ",{"tag":192,"to":347,"children":348},"\u002Fblog\u002Fllm-cost-latency-prompt-caching-routing",[349],"prompt caching and model routing",{"type":351,"variant":352,"title":353,"body":354},"callout","tip","Keep volatile text out of the prefix",[355],[356],"Dates, user names, feature flags and A\u002FB variant text belong at the end of the prompt, not in the system prompt. Otherwise every variant and every day is a separate cache prefix, and a harmless experiment doubles your input cost.",{"type":133,"level":134,"id":119,"text":120},{"type":126,"content":359},[360,361,363],"For the projects I build, the default is git first. Prompts live in a ",{"tag":184,"children":362},[291]," module with their model, parameters and schema; a version string is bumped on every change; a promptfoo or equivalent eval runs on the pull request; and the version goes into every trace. A registry comes in when someone outside the dev team needs to edit prompts, or when a rollback has to be faster than a deploy. Then git stays the source of truth and CI publishes to the registry, so the label moves only after the eval gate passes.",{"type":365,"items":366},"list",[367,373,378,383],[368,372],{"tag":369,"children":370},"strong",[371],"Never edit production text in place."," Create a version, evaluate it, move the label.",[374,377],{"tag":369,"children":375},[376],"Pin versions in evals and batch jobs",", follow labels only in interactive paths.",[379,382],{"tag":369,"children":380},[381],"Keep a fallback prompt in the repository"," for every registry-managed prompt.",[384,387],{"tag":369,"children":385},[386],"Treat prompts as code in reviews:"," the diff of a prompt deserves the same attention as the diff of a SQL query.",{"type":133,"level":134,"id":122,"text":123},{"type":365,"ordered":390,"items":391},true,[392,396,399,402,405,408,411,414],[393],{"tag":394,"href":38,"children":395},"a",[37],[397],{"tag":394,"href":41,"children":398},[40],[400],{"tag":394,"href":44,"children":401},[43],[403],{"tag":394,"href":47,"children":404},[46],[406],{"tag":394,"href":50,"children":407},[49],[409],{"tag":394,"href":53,"children":410},[52],[412],{"tag":394,"href":56,"children":413},[55],[415],{"tag":394,"href":59,"children":416},[58],[418,486,542,595],{"slug":419,"published":5,"minutes":420,"category":7,"tags":421,"keywords":427,"about":439,"sources":449,"cover":480,"og":481,"expertise":62,"locales":482,"lang":64,"title":483,"description":484,"coverAlt":485},"llm-rate-limits-retries-provider-fallback",10,[422,423,424,425,426],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[428,429,430,431,432,433,434,435,436,437,438],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[440,443,446],{"name":441,"url":442},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":444,"url":445},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":447,"url":448},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[450,453,456,459,462,465,468,471,474,477],{"title":451,"url":452},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":454,"url":455},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":457,"url":458},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":460,"url":461},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":463,"url":464},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":466,"url":467},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":469,"url":470},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":472,"url":473},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":475,"url":476},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":478,"url":479},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[64,65,66],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":487,"published":5,"minutes":488,"category":7,"tags":489,"keywords":494,"about":505,"sources":515,"cover":536,"og":537,"expertise":62,"locales":538,"lang":64,"title":539,"description":540,"coverAlt":541},"llm-batch-api-offline-workloads",9,[490,491,11,492,493],"Batch API","Cost","Prompt caching","Data pipelines",[495,496,497,498,499,500,501,502,503,504],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[506,509,512],{"name":507,"url":508},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":510,"url":511},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":513,"url":514},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[516,519,521,522,525,528,530,533],{"title":517,"url":518},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":520,"url":452},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":58,"url":59},{"title":523,"url":524},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":526,"url":527},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":529,"url":458},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":531,"url":532},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":534,"url":535},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[64,65,66],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",{"slug":543,"published":544,"minutes":488,"category":7,"tags":545,"keywords":551,"about":563,"sources":570,"cover":589,"og":590,"expertise":62,"locales":591,"lang":64,"title":592,"description":593,"coverAlt":594},"structured-outputs-json-schema-llm","2026-10-16",[546,547,548,549,550],"Structured outputs","JSON Schema","Constrained decoding","LLM APIs","Validation",[552,553,554,555,556,557,558,559,560,561,562],"structured outputs LLM","JSON schema LLM output","OpenAI structured outputs strict mode","Claude structured outputs","Gemini structured output","constrained decoding","JSON mode vs structured outputs","validate LLM JSON output","Zod structured output","Pydantic LLM output","vLLM structured outputs",[564,567,569],{"name":565,"url":566},"JSON","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FJSON",{"name":547,"url":568},"https:\u002F\u002Fjson-schema.org\u002F",{"name":513,"url":514},[571,574,577,580,583,586],{"title":572,"url":573},"OpenAI API docs: Structured model outputs","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fstructured-outputs",{"title":575,"url":576},"Claude API docs: Structured outputs","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fstructured-outputs",{"title":578,"url":579},"Gemini API docs: Structured outputs","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fstructured-output",{"title":581,"url":582},"vLLM docs: Structured Outputs","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fstructured_outputs.html",{"title":584,"url":585},"Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.02442",{"title":587,"url":588},"Geng et al.: JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.10868","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fog.jpg",[64,65,66],"Structured outputs with JSON Schema: what the guarantee covers, and what it does not","How structured outputs and constrained decoding work on OpenAI, Claude, Gemini and vLLM, which schema rules matter, and what you still validate.","A fixed four-compartment frame, three blocks seated, one too-wide block hovering above.",{"slug":596,"published":544,"minutes":488,"category":7,"tags":597,"keywords":603,"about":616,"sources":624,"cover":646,"og":647,"expertise":62,"locales":648,"lang":64,"title":649,"description":650,"coverAlt":651},"evaluating-ai-agents-trajectories",[598,599,600,601,602],"Agent evals","Trajectories","Tool calling","pass^k","LLM-as-judge",[604,605,606,607,608,609,610,611,612,613,614,615],"evaluating AI agents","agent trajectory evaluation","agent evals","tool call accuracy","trajectory match evaluator","pass^k metric","tau-bench","LLM as a judge for agents","agent regression testing","outcome vs trajectory evaluation","agentevals","tool correctness metric",[617,620,623],{"name":618,"url":619},"Intelligent agent","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIntelligent_agent",{"name":621,"url":622},"Software testing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSoftware_testing",{"name":513,"url":514},[625,628,631,634,637,640,643],{"title":626,"url":627},"Anthropic: Demystifying evals for AI agents (Jan 2026)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents",{"title":629,"url":630},"Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045",{"title":632,"url":633},"Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07982",{"title":635,"url":636},"LangChain: agentevals (GitHub)","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals",{"title":638,"url":639},"DeepEval docs: Tool Correctness","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fmetrics-tool-correctness",{"title":641,"url":642},"Ragas docs: Agentic or tool use metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002Fagents\u002F",{"title":644,"url":645},"OpenAI API docs: Trace grading","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ftrace-grading","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fog.jpg",[64,65,66],"Evaluating AI agents: grading trajectories, outcomes and reliability","How to evaluate AI agents: outcome vs trajectory graders, trajectory match modes, pass^k, isolated environments and LLM judges that hold up.","Two dotted paths of circles between two squares, a wide bracket framing the path that bends.",1792222776042]