[{"data":1,"prerenderedAt":810},["ShallowReactive",2],{"blog-llm-rate-limits-retries-provider-fallback-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":26,"sources":36,"cover":67,"og":68,"expertise":69,"locales":70,"lang":71,"title":74,"description":75,"coverAlt":76,"metaTitle":77,"takeaways":78,"faq":84,"toc":100,"blocks":131,"others":582},"llm-rate-limits-retries-provider-fallback","2026-10-17",10,"llmops",[9,10,11,12,13],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[15,16,17,18,19,20,21,22,23,24,25],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[27,30,33],{"name":28,"url":29},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":31,"url":32},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":34,"url":35},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[37,40,43,46,49,52,55,58,61,64],{"title":38,"url":39},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":41,"url":42},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":44,"url":45},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":47,"url":48},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":50,"url":51},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":53,"url":54},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":56,"url":57},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":59,"url":60},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":62,"url":63},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":65,"url":66},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg","ai-engineer",[71,72,73],"en","de","hu","LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.","LLM rate limits: retries and fallback · Balázs Csorba",[79,80,81,82,83],"Provider limits are token buckets measured per minute in requests and tokens, but enforced over shorter intervals, so bursts fail even when the minute total looks fine.","Classify errors before retrying: 429 with retry-after and 5xx are transient, a spend-cap 429, 400s and auth errors are not.","Use exponential backoff with jitter, honour retry-after as a minimum, and cap both attempts and total time.","Retry in exactly one layer. SDK, gateway and app retries multiply, and the AWS example shows how fast that grows.","A fallback model is a different product: evaluate it on your own test set before it ever serves production traffic.",[85,88,91,94,97],{"q":86,"a":87},"What is the difference between a 429 and a 529 from the Claude API?","A 429 rate_limit_error means your organization hit one of its own limits, such as requests or tokens per minute, or its monthly spend cap. A 529 overloaded_error means the API is temporarily overloaded across all users. Both are usually worth a delayed retry, except the spend-cap 429, which has no retry-after header and keeps failing until access resumes.",{"q":89,"a":90},"How many times should I retry an LLM request?","Two to four attempts with exponential backoff is a common range, and both the Anthropic SDK and the Vercel AI SDK retry twice by default. More important than the count is a cap on total wall time, so a user request does not hang for minutes, and making sure only one layer of your stack retries.",{"q":92,"a":93},"Does max_tokens count against rate limits?","It depends on the provider. Anthropic says max_tokens does not factor into its output-token limit, which counts only generated tokens. OpenAI calculates the request against the limit as the maximum of max_tokens and the estimated token count, so an oversized max_tokens uses up limit headroom.",{"q":95,"a":96},"Should I use LiteLLM, OpenRouter or my own code for fallbacks?","If you already run a gateway, configure fallbacks there and turn off retries elsewhere. LiteLLM gives you per-error retry policies, cooldowns and context-window fallbacks you host yourself; OpenRouter gives you a models array behind one API key. Own code is fine for one or two providers, but you will end up rebuilding cooldowns.",{"q":98,"a":99},"Is falling back to another provider safe for structured output?","Only if you have tested it. Providers differ in how they enforce JSON Schema, which features they support and how they phrase refusals. Run the same evaluation set against the fallback model and validate every response against your schema regardless of which model produced it.",[101,104,107,110,113,116,119,122,125,128],{"id":102,"title":103},"how-limits-work","How provider rate limits actually work",{"id":105,"title":106},"classify-errors","Classify the error before you retry",{"id":108,"title":109},"backoff-jitter","Exponential backoff with jitter, and a deadline",{"id":111,"title":112},"retry-one-layer","Retry in exactly one layer",{"id":114,"title":115},"fallback-chain","Building a fallback chain",{"id":117,"title":118},"fallback-pitfalls","Why fallbacks silently change your product",{"id":120,"title":121},"circuit-breakers","Cooldowns and circuit breakers",{"id":123,"title":124},"raise-the-ceiling","Raise the ceiling before you hit it",{"id":126,"title":127},"timeouts-streaming","Timeouts and long requests",{"id":129,"title":130},"sources","Sources",[132,136,139,142,145,156,218,221,222,225,284,305,308,309,315,317,320,321,335,363,364,367,376,413,415,416,419,460,461,472,475,476,484,517,518,535,547,548],{"type":133,"content":134},"paragraph",[135],"Every LLM feature that reaches real traffic eventually meets a 429. Sometimes it is your own fault, a batch job that shares a key with the chat endpoint. Sometimes it is the provider, overloaded at peak hours. Either way the user sees a spinner, then an error, and the usual reaction is to wrap the call in a retry loop and move on. That fixes the demo and creates the next outage.",{"type":133,"content":137},[138],"This guide covers what actually happens behind a rate limit, which errors deserve a retry and which never do, how to back off without synchronising every client, why retries belong in exactly one layer, and how to build a fallback chain to another deployment or provider without silently changing what your feature returns. The numbers come from the Anthropic and OpenAI documentation as of October 2026.",{"type":140,"level":141,"id":102,"text":103},"heading",2,{"type":133,"content":143},[144],"Anthropic measures Messages API limits in requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM), per model class, and uses a token bucket: capacity refills continuously up to the maximum instead of resetting at the top of each minute. The documentation adds a detail that explains many puzzling 429s: a limit of 60 RPM may be enforced as one request per second, so a short burst fails even if the minute as a whole stays under the limit.",{"type":133,"content":146},[147,148,152,153,155],"OpenAI counts requests and tokens per minute and per day, plus images per minute, and whichever is hit first triggers the error. Two further details matter for design. OpenAI counts a request against the token limit as the maximum of ",{"tag":149,"children":150},"code",[151],"max_tokens"," and its estimate of the request size, so a generous ",{"tag":149,"children":154},[151]," eats headroom; Anthropic says the opposite for its output limit, which counts only generated tokens. And OpenAI notes that unsuccessful requests still count toward the per-minute limit, which is why tight retry loops make a 429 worse rather than better.",{"type":157,"head":158,"rows":165},"table",[159,161,163],[160],"Aspect",[162],"Anthropic Claude API",[164],"OpenAI API",[166,173,180,186,204,211],[167,169,171],[168],"Dimensions",[170],"RPM, ITPM, OTPM",[172],"RPM, RPD, TPM, TPD, IPM",[174,176,178],[175],"Cached input",[177],"Cache reads do not count toward ITPM on most models",[179],"Counted in TPM",[181,182,184],[151],[183],"Not counted; OTPM uses generated tokens",[185],"Counted as max(max_tokens, estimate)",[187,189,197],[188],"Headers",[190,193,194],{"tag":149,"children":191},[192],"anthropic-ratelimit-*",", ",{"tag":149,"children":195},[196],"retry-after",[198,193,201],{"tag":149,"children":199},[200],"x-ratelimit-*",{"tag":149,"children":202},[203],"Retry-After",[205,207,209],[206],"Retry-After sent on",[208],"Most 429s, not the spend-cap 429",[210],"429 and 503 responses",[212,214,216],[213],"Burst protection",[215],"Acceleration limits on sharp usage increases",[217],"Shorter enforcement windows",{"type":133,"content":219},[220],"Anthropic also applies acceleration limits: if your organization ramps up usage sharply, you can get 429s even below your tier limits, and the advice is to ramp traffic gradually. That matters on launch day and when a backfill job starts at full speed.",{"type":140,"level":141,"id":105,"text":106},{"type":133,"content":223},[224],"A retry is only useful when the next attempt can succeed without anything changing. Most retry bugs come from treating every non-200 the same. The Claude API error list makes the split fairly clean:",{"type":157,"head":226,"rows":233},[227,229,231],[228],"Response",[230],"Meaning",[232],"What to do",[234,243,253,267,277],[235,239,241],[236,237],"429 + ",{"tag":149,"children":238},[196],[240],"A rate limit was hit",[242],"Wait at least retry-after, then retry",[244,249,251],[245,246],"429 without retry-after, ",{"tag":149,"children":247},[248],"enforced_spend_limit_reached",[250],"Monthly spend cap of the tier reached",[252],"Do not retry; alert a human or fail over",[254,263,265],[255,256,259,260],"500 ",{"tag":149,"children":257},[258],"api_error",", 529 ",{"tag":149,"children":261},[262],"overloaded_error",[264],"Transient server-side problem",[266],"Back off and retry, then fall back",[268,273,275],[269,270],"504 ",{"tag":149,"children":271},[272],"timeout_error",[274],"Request timed out while processing",[276],"Retry once; stream long requests",[278,280,282],[279],"400, 401, 403, 404, 413",[281],"The request itself is wrong, or a self-set spend limit (400)",[283],"Never retry unchanged; log and fix",{"type":285,"variant":286,"title":287,"body":288},"callout","warn","Not every 429 is temporary",[289],[290,291,294,295,297,298,301,302,304],"When an Anthropic organization reaches its tier spend cap, the API returns a 429 with error type ",{"tag":149,"children":292},[293],"rate_limit_error",", no ",{"tag":149,"children":296},[196]," header and ",{"tag":149,"children":299},[300],"error.details.error_code"," set to ",{"tag":149,"children":303},[248],". Usage stays paused until the first day of the next month at 00:00 UTC. A generic retry loop, including the SDK default, keeps hammering a wall. Detect that code, stop retrying and page someone.",{"type":133,"content":306},[307],"Streaming adds one more case. With server-sent events an error can arrive after the API has already returned 200, so status-code based retry logic never sees it. Whether to retry a half-streamed answer is a product decision: for a chat UI, show what arrived and offer a retry button; for a background job, discard the partial output and retry the whole request.",{"type":140,"level":141,"id":108,"text":109},{"type":133,"content":310},[311,312,314],"The OpenAI guide and the AWS Builders Library agree on the shape: wait a little, then longer after each failure, add randomness so clients do not retry in lockstep, and stop after a limit. OpenAI adds that ",{"tag":149,"children":313},[203]," should be treated as a minimum, not a suggestion. Per HTTP semantics the header holds either a number of seconds or an HTTP date, so parse both.",{"type":149,"code":316},"\u002F\u002F One retry wrapper for the whole app. Turn SDK retries off (maxRetries: 0) when you use it.\nconst RETRYABLE = new Set([408, 429, 500, 502, 503, 504, 529])\n\nfunction retryAfterMs(h: Headers): number | null {\n  const v = h.get('retry-after')\n  if (!v) return null\n  const s = Number(v)\n  if (!Number.isNaN(s)) return s * 1000\n  const t = Date.parse(v)              \u002F\u002F HTTP-date form\n  return Number.isNaN(t) ? null : Math.max(0, t - Date.now())\n}\n\nexport async function withRetry\u003CT>(call: () => Promise\u003CResponse>, parse: (r: Response) => Promise\u003CT>,\n  { attempts = 4, baseMs = 500, capMs = 20_000, deadlineMs = 45_000 } = {}): Promise\u003CT> {\n  const start = Date.now()\n  for (let i = 0; ; i++) {\n    const res = await call()\n    if (res.ok) return parse(res)\n    const body = await res.clone().json().catch(() => ({}))\n    const spendCap = body?.error?.details?.error_code === 'enforced_spend_limit_reached'\n    if (!RETRYABLE.has(res.status) || spendCap || i + 1 >= attempts) throw new Error(`LLM ${res.status}`)\n    const backoff = Math.random() * Math.min(capMs, baseMs * 2 ** i)   \u002F\u002F \"full jitter\"\n    const wait = Math.max(retryAfterMs(res.headers) ?? 0, backoff)\n    if (Date.now() - start + wait > deadlineMs) throw new Error(`LLM ${res.status}: retry budget spent`)\n    await new Promise((r) => setTimeout(r, wait))\n  }\n}",{"type":133,"content":318},[319],"The deadline matters as much as the attempt count. A chat request that is still retrying after 45 seconds is a failure whatever happens next, so give up early and let the fallback chain or the UI handle it. Background jobs can afford longer deadlines, but they should also back off much harder, because they compete with your interactive traffic for the same limits.",{"type":140,"level":141,"id":111,"text":112},{"type":133,"content":322},[323,324,326,327,330,331,334],"The AWS article has the clearest warning: with a five-deep call stack where each layer retries three times, the load on the bottom dependency can grow 243 times. LLM stacks hide the same trap. The Anthropic SDK retries connection errors, 429s and 5xx twice by default and honours ",{"tag":149,"children":325},[196],". The Vercel AI SDK has ",{"tag":149,"children":328},[329],"maxRetries"," with a default of 2. A gateway such as LiteLLM has ",{"tag":149,"children":332},[333],"num_retries",". If your own code also loops, one user click can turn into dozens of upstream calls during an incident, exactly when the provider is least able to absorb them.",{"type":336,"items":337},"list",[338,344,353,358],[339,343],{"tag":340,"children":341},"strong",[342],"Pick the layer that knows the most."," Usually the gateway, if you run one, because it sees all traffic and every deployment. Otherwise your own wrapper.",[345,348,349,352],{"tag":340,"children":346},[347],"Set retries to zero everywhere else."," ",{"tag":149,"children":350},[351],"maxRetries: 0"," in the SDK client, no loops in route handlers.",[354,357],{"tag":340,"children":355},[356],"Add a retry budget."," AWS limits local retries with a token bucket: retries flow freely while tokens last, then drop to a fixed rate. That keeps an outage from doubling your request volume.",[359,362],{"tag":340,"children":360},[361],"Make side effects idempotent first."," An agent step that sends an email or writes an order must not run twice because the model call after it was retried.",{"type":140,"level":141,"id":114,"text":115},{"type":133,"content":365},[366],"Retries help with blips. When a model stays unavailable or your limit for it is exhausted, you need somewhere else to send the request. I order the options by how much they change the output: the same model on another deployment or region first, a sibling model from the same provider next, a different provider last, and a graceful degradation (cached answer, simpler feature, honest error) at the end.",{"type":368,"attrs":369,"inner":373,"caption":374},"diagram",{"viewBox":370,"role":371,"aria-labelledby":372},"0 0 720 170","img","rlf1-t rlf1-d","\u003Ctitle id=\"rlf1-t\">Fallback chain ordered by how much the output changes\u003C\u002Ftitle>\u003Cdesc id=\"rlf1-d\">A request first retries with backoff, then moves to the same model on another deployment, then a sibling model, then another provider, and finally degrades gracefully.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"rlf1-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"20\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"79\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Retry\u003C\u002Ftext>\u003Ctext x=\"79\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">backoff + jitter\u003C\u002Ftext>\u003Cpath d=\"M138 85H158\" class=\"d-line\" marker-end=\"url(#rlf1-ah)\" \u002F>\u003Crect x=\"160\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"220\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Same model\u003C\u002Ftext>\u003Ctext x=\"220\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">other region\u003C\u002Ftext>\u003Cpath d=\"M279 85H299\" class=\"d-line\" marker-end=\"url(#rlf1-ah)\" \u002F>\u003Crect x=\"301\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"360\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Sibling\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">same provider\u003C\u002Ftext>\u003Cpath d=\"M419 85H439\" class=\"d-line\" marker-end=\"url(#rlf1-ah)\" \u002F>\u003Crect x=\"441\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"500\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Other provider\u003C\u002Ftext>\u003Ctext x=\"500\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">evaluated first\u003C\u002Ftext>\u003Cpath d=\"M560 85H580\" class=\"d-line\" marker-end=\"url(#rlf1-ah)\" \u002F>\u003Crect x=\"582\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"641\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Degrade\u003C\u002Ftext>\u003Ctext x=\"641\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">honest error\u003C\u002Ftext>",[375],"The further right a request lands, the more its output may differ from what you tested.",{"type":133,"content":377},[378,379,384,385,388,389,392,393,395,396,399,400,403,404,408,409,412],"You rarely need to write this yourself. ",{"tag":380,"to":381,"children":382},"link","\u002Ftools\u002Flitellm",[383],"LiteLLM"," takes a ",{"tag":149,"children":386},[387],"fallbacks"," list in ",{"tag":149,"children":390},[391],"router_settings"," that maps a model group to ordered backups and fires after ",{"tag":149,"children":394},[333]," attempts. It also has ",{"tag":149,"children":397},[398],"context_window_fallbacks"," for prompts that are too long and ",{"tag":149,"children":401},[402],"content_policy_fallbacks"," for refusals by a provider filter. ",{"tag":380,"to":405,"children":406},"\u002Ftools\u002Fopenrouter",[407],"OpenRouter"," accepts a ",{"tag":149,"children":410},[411],"models"," array in priority order, tries the next model when one returns an error, including rate limits, downtime and context-length errors, and bills you for the model that actually handled the request.",{"type":149,"code":414},"# LiteLLM proxy: retries live here, nowhere else\nrouter_settings:\n  num_retries: 2\n  allowed_fails: 3        # failures per minute before a deployment cools down\n  cooldown_time: 30       # seconds the deployment is skipped\n  retry_policy:\n    RateLimitErrorRetries: 3\n    BadRequestErrorRetries: 0\n    AuthenticationErrorRetries: 0\n    DefaultRetries: 2\n  fallbacks: [{\"support-primary\": [\"support-secondary\"]}]\n  enable_pre_call_checks: true   # needed for context-window fallbacks to check input size",{"type":140,"level":141,"id":117,"text":118},{"type":133,"content":417},[418],"A fallback that returns a 200 can still be a failure. Prompts tuned for one model often behave differently on another; structured output support and JSON Schema coverage vary between providers; tool-calling formats and refusal styles differ; and prompt caches are per provider, so the first minutes on a fallback cost more and run slower. Teams discover this when the incident is over and support tickets mention odd answers.",{"type":336,"items":420},[421,431,436,446,451],[422,425,426,430],{"tag":340,"children":423},[424],"Evaluate the fallback like a migration."," Run your regression set against every model in the chain, as described in ",{"tag":380,"to":427,"children":428},"\u002Fblog\u002Fllm-model-migration-regression-testing",[429],"LLM model migrations without regressions",".",[432,435],{"tag":340,"children":433},[434],"Keep a prompt per model."," A fallback entry is a model plus its prompt and parameters, not just a model name.",[437,440,441,445],{"tag":340,"children":438},[439],"Validate output regardless of source."," Schema validation and business checks run on every response, which ",{"tag":380,"to":442,"children":443},"\u002Fblog\u002Fstructured-outputs-json-schema-llm",[444],"structured outputs"," do not make optional.",[447,450],{"tag":340,"children":448},[449],"Record which model answered."," Put the serving model in your traces so quality dashboards do not blend two products into one line.",[452,455,456,430],{"tag":340,"children":453},[454],"Check data residency."," Falling back from an EU region to a US endpoint can break the promise in your DPA; see ",{"tag":380,"to":457,"children":458},"\u002Fblog\u002Fgdpr-llm-api-eu-data-residency",[459],"GDPR LLM data residency",{"type":140,"level":141,"id":120,"text":121},{"type":133,"content":462},[463,464,467,468,471],"Without memory, every request rediscovers that a deployment is down and pays the timeout. Martin Fowler's circuit breaker fixes that with three states: closed (calls pass and failures are counted), open (calls fail immediately without touching the dependency) and half-open (after a wait, one trial call decides whether to close again). LiteLLM ships a simpler version per deployment: after ",{"tag":149,"children":465},[466],"allowed_fails"," failures in a minute (default 3) a deployment is cooled down for ",{"tag":149,"children":469},[470],"cooldown_time"," seconds (default 5) and traffic goes to the others.",{"type":133,"content":473},[474],"The AWS article is more cautious about breakers: they add modal behaviour that is hard to test and can slow recovery, which is why AWS prefers local retry budgets. My compromise is short cooldowns per deployment, a retry budget per client, and no global breaker that can take a whole feature offline because of one noisy minute.",{"type":140,"level":141,"id":123,"text":124},{"type":133,"content":477},[478,479,483],"The cheapest 429 is the one that never happens. On most Claude models, cache reads do not count toward ITPM, so caching raises effective throughput: Anthropic's example is a 2,000,000 ITPM limit with an 80% cache hit rate handling 10,000,000 total input tokens per minute. The setup is described in ",{"tag":380,"to":480,"children":481},"\u002Fblog\u002Fllm-cost-latency-prompt-caching-routing",[482],"prompt caching and model routing",". A few more levers:",{"type":336,"items":485},[486,491,496,508],[487,490],{"tag":340,"children":488},[489],"Separate workloads."," Anthropic workspaces can have their own lower limits, so a backfill cannot starve the chat endpoint. Separate keys or projects do the same elsewhere.",[492,495],{"tag":340,"children":493},[494],"Move offline work off the synchronous API."," Batch endpoints have their own limits and do not consume the Messages API limits.",[497,348,500,503,504,507],{"tag":340,"children":498},[499],"Read the headers.",{"tag":149,"children":501},[502],"anthropic-ratelimit-*-remaining"," and ",{"tag":149,"children":505},[506],"x-ratelimit-remaining-*"," let a client throttle itself before the provider does.",[509,512,513,430],{"tag":340,"children":510},[511],"Ramp up gradually."," Start backfills slowly to stay clear of acceleration limits, and protect public endpoints from abuse, see ",{"tag":380,"to":514,"children":515},"\u002Fblog\u002Fllm-endpoint-abuse-denial-of-wallet",[516],"denial of wallet",{"type":140,"level":141,"id":126,"text":127},{"type":133,"content":519},[520,521,524,525,193,528,503,531,534],"A retry policy without a timeout is incomplete, because a hung connection never produces an error to retry. AWS suggests deriving timeouts from downstream latency percentiles rather than guessing. For LLM calls that means separate budgets for time to first token and total duration. Anthropic recommends streaming or the Batches API for requests that may run past ten minutes, since some networks drop idle connections, and its SDKs refuse non-streaming requests that are expected to exceed a ten-minute timeout. The AI SDK accepts a ",{"tag":149,"children":522},[523],"timeout"," object with ",{"tag":149,"children":526},[527],"totalMs",{"tag":149,"children":529},[530],"stepMs",{"tag":149,"children":532},[533],"chunkMs",", which maps neatly onto those budgets.",{"type":133,"content":536},[537,538,541,542,546],"Log the ",{"tag":149,"children":539},[540],"request-id"," header with every failure. When an incident is the provider's, that id is what their support asks for, and it turns a vague \"the API was slow\" into something they can trace. Put retries, fallbacks and the serving model into the same trace, as in ",{"tag":380,"to":543,"children":544},"\u002Fblog\u002Fagent-observability-opentelemetry",[545],"observability for LLM agents",", and you can see afterwards how much of an incident your chain absorbed.",{"type":140,"level":141,"id":129,"text":130},{"type":336,"ordered":549,"items":550},true,[551,555,558,561,564,567,570,573,576,579],[552],{"tag":553,"href":39,"children":554},"a",[38],[556],{"tag":553,"href":42,"children":557},[41],[559],{"tag":553,"href":45,"children":560},[44],[562],{"tag":553,"href":48,"children":563},[47],[565],{"tag":553,"href":51,"children":566},[50],[568],{"tag":553,"href":54,"children":569},[53],[571],{"tag":553,"href":57,"children":572},[56],[574],{"tag":553,"href":60,"children":575},[59],[577],{"tag":553,"href":63,"children":578},[62],[580],{"tag":553,"href":66,"children":581},[65],[583,644,700,753],{"slug":584,"published":5,"minutes":585,"category":7,"tags":586,"keywords":592,"about":603,"sources":613,"cover":638,"og":639,"expertise":69,"locales":640,"lang":71,"title":641,"description":642,"coverAlt":643},"prompt-versioning-management",8,[587,588,589,590,591],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[593,594,595,596,597,598,599,600,601,602],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[604,607,610],{"name":605,"url":606},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":608,"url":609},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":611,"url":612},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[614,617,620,623,626,629,632,635],{"title":615,"url":616},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":618,"url":619},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":621,"url":622},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":624,"url":625},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":627,"url":628},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":630,"url":631},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":633,"url":634},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":636,"url":637},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[71,72,73],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":645,"published":5,"minutes":646,"category":7,"tags":647,"keywords":652,"about":663,"sources":673,"cover":694,"og":695,"expertise":69,"locales":696,"lang":71,"title":697,"description":698,"coverAlt":699},"llm-batch-api-offline-workloads",9,[648,649,589,650,651],"Batch API","Cost","Prompt caching","Data pipelines",[653,654,655,656,657,658,659,660,661,662],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[664,667,670],{"name":665,"url":666},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":668,"url":669},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":671,"url":672},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[674,677,679,680,683,686,688,691],{"title":675,"url":676},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":678,"url":39},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":636,"url":637},{"title":681,"url":682},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":684,"url":685},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":687,"url":45},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":689,"url":690},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":692,"url":693},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[71,72,73],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",{"slug":701,"published":702,"minutes":646,"category":7,"tags":703,"keywords":709,"about":721,"sources":728,"cover":747,"og":748,"expertise":69,"locales":749,"lang":71,"title":750,"description":751,"coverAlt":752},"structured-outputs-json-schema-llm","2026-10-16",[704,705,706,707,708],"Structured outputs","JSON Schema","Constrained decoding","LLM APIs","Validation",[710,711,712,713,714,715,716,717,718,719,720],"structured outputs LLM","JSON schema LLM output","OpenAI structured outputs strict mode","Claude structured outputs","Gemini structured output","constrained decoding","JSON mode vs structured outputs","validate LLM JSON output","Zod structured output","Pydantic LLM output","vLLM structured outputs",[722,725,727],{"name":723,"url":724},"JSON","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FJSON",{"name":705,"url":726},"https:\u002F\u002Fjson-schema.org\u002F",{"name":671,"url":672},[729,732,735,738,741,744],{"title":730,"url":731},"OpenAI API docs: Structured model outputs","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fstructured-outputs",{"title":733,"url":734},"Claude API docs: Structured outputs","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fstructured-outputs",{"title":736,"url":737},"Gemini API docs: Structured outputs","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fstructured-output",{"title":739,"url":740},"vLLM docs: Structured Outputs","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fstructured_outputs.html",{"title":742,"url":743},"Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.02442",{"title":745,"url":746},"Geng et al.: JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.10868","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fog.jpg",[71,72,73],"Structured outputs with JSON Schema: what the guarantee covers, and what it does not","How structured outputs and constrained decoding work on OpenAI, Claude, Gemini and vLLM, which schema rules matter, and what you still validate.","A fixed four-compartment frame, three blocks seated, one too-wide block hovering above.",{"slug":754,"published":702,"minutes":646,"category":7,"tags":755,"keywords":761,"about":774,"sources":782,"cover":804,"og":805,"expertise":69,"locales":806,"lang":71,"title":807,"description":808,"coverAlt":809},"evaluating-ai-agents-trajectories",[756,757,758,759,760],"Agent evals","Trajectories","Tool calling","pass^k","LLM-as-judge",[762,763,764,765,766,767,768,769,770,771,772,773],"evaluating AI agents","agent trajectory evaluation","agent evals","tool call accuracy","trajectory match evaluator","pass^k metric","tau-bench","LLM as a judge for agents","agent regression testing","outcome vs trajectory evaluation","agentevals","tool correctness metric",[775,778,781],{"name":776,"url":777},"Intelligent agent","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIntelligent_agent",{"name":779,"url":780},"Software testing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSoftware_testing",{"name":671,"url":672},[783,786,789,792,795,798,801],{"title":784,"url":785},"Anthropic: Demystifying evals for AI agents (Jan 2026)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents",{"title":787,"url":788},"Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045",{"title":790,"url":791},"Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07982",{"title":793,"url":794},"LangChain: agentevals (GitHub)","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals",{"title":796,"url":797},"DeepEval docs: Tool Correctness","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fmetrics-tool-correctness",{"title":799,"url":800},"Ragas docs: Agentic or tool use metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002Fagents\u002F",{"title":802,"url":803},"OpenAI API docs: Trace grading","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ftrace-grading","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fog.jpg",[71,72,73],"Evaluating AI agents: grading trajectories, outcomes and reliability","How to evaluate AI agents: outcome vs trajectory graders, trajectory match modes, pass^k, isolated environments and LLM judges that hold up.","Two dotted paths of circles between two squares, a wide bracket framing the path that bends.",1792222776034]