[{"data":1,"prerenderedAt":782},["ShallowReactive",2],{"blog-llm-batch-api-offline-workloads-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":25,"sources":35,"cover":60,"og":61,"expertise":62,"locales":63,"lang":64,"title":67,"description":68,"coverAlt":69,"metaTitle":70,"takeaways":71,"faq":77,"toc":93,"blocks":124,"others":547},"llm-batch-api-offline-workloads","2026-10-17",9,"llmops",[9,10,11,12,13],"Batch API","Cost","LLMOps","Prompt caching","Data pipelines",[15,16,17,18,19,20,21,22,23,24],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[26,29,32],{"name":27,"url":28},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":30,"url":31},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":33,"url":34},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[36,39,42,45,48,51,54,57],{"title":37,"url":38},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":40,"url":41},"Claude Platform Docs: Rate limits (Message Batches API)","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":43,"url":44},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching",{"title":46,"url":47},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":49,"url":50},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":52,"url":53},"OpenAI API Docs: Rate limits (batch queue limits)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":55,"url":56},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":58,"url":59},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg","ai-engineer",[64,65,66],"en","de","hu","LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.","LLM batch APIs: half-price offline work · Balázs Csorba",[72,73,74,75,76],"Anthropic, OpenAI and Gemini all charge 50% of standard prices for batch requests, in exchange for asynchronous results within a 24-hour window.","Results come back in any order: always join them to source records by custom_id or key, never by position.","Batch limits are a separate pool from synchronous limits, so moving backfills and evals to batch also protects your interactive traffic.","Batch and prompt caching discounts stack, but batch cache hits are best effort; use identical cache_control blocks and consider the 1-hour cache.","Batch data sits on provider storage for weeks: Anthropic keeps results 29 days, OpenAI deletes output files after 30 days, Gemini after 6 weeks.",[78,81,84,87,90],{"q":79,"a":80},"How much cheaper is a batch API?","Anthropic, OpenAI and Google Gemini all price batch requests at 50% of standard rates. On the Claude API that applies to input, output and special tokens, so Claude Opus 5.5 costs 2 dollars per million input tokens and 10 dollars per million output tokens in a batch. Prompt caching discounts apply on top.",{"q":82,"a":83},"How long does a batch take?","All three providers work with a 24-hour window. Anthropic says most batches finish in less than an hour, but requests that are not processed within 24 hours expire. Plan pipelines around the 24-hour worst case, not the typical case.",{"q":85,"a":86},"Can I use tools and structured output in batch requests?","On the Claude API, almost any Messages request can be batched, including tool use, server tools, vision, system prompts and multi-turn conversations; stream, speed and max_tokens 0 are not supported. Client tools still need a round trip, so each tool call means a new request. On Bedrock, tool calling needs the InvokeModel format rather than Converse.",{"q":88,"a":89},"What happens to failed requests in a batch?","Each request gets its own result. On the Claude API, errored, canceled and expired requests are not billed. On OpenAI, failures go to a separate error file, and when a batch expires you pay for the requests that did complete. Collect the failures and resubmit them in a new batch.",{"q":91,"a":92},"Batch API or OpenAI flex processing?","Flex is a synchronous request with service_tier set to flex, priced at batch rates but slower and sometimes unavailable, in which case it returns a 429 you are not charged for. It suits jobs that want simple request-response code. A batch suits large volumes where you can wait for a file of results.",[94,97,100,103,106,109,112,115,118,121],{"id":95,"title":96},"when-batch-fits","Which workloads belong in a batch",{"id":98,"title":99},"provider-comparison","Anthropic, OpenAI, Gemini and Bedrock compared",{"id":101,"title":102},"batch-lifecycle","The batch lifecycle",{"id":104,"title":105},"building-requests","Building requests you can match and retry",{"id":107,"title":108},"results-and-failures","Partial failures and what you pay for",{"id":110,"title":111},"caching-with-batches","Stacking batch and prompt caching discounts",{"id":113,"title":114},"limits-throughput","Limits, throughput and spend",{"id":116,"title":117},"flex-alternative","When flex processing is the simpler choice",{"id":119,"title":120},"data-retention","Data retention and GDPR",{"id":122,"title":123},"sources","Sources",[125,129,132,135,138,181,184,185,271,286,287,296,341,342,352,354,361,362,381,411,412,419,426,427,430,437,447,448,463,497,498,505,518,519],{"type":126,"content":127},"paragraph",[128],"A surprising share of LLM spend never touches a user. Re-classifying a support archive after a taxonomy change, writing product descriptions for a catalogue import, running a nightly eval suite, re-embedding documents after a model switch: none of these needs an answer in two seconds, yet many teams push them through the same synchronous endpoint as their chat feature, at full price and competing for the same rate limits.",{"type":126,"content":130},[131],"Batch APIs exist for exactly this work. You submit many requests at once, the provider processes them when it has capacity, and you collect a file of results within 24 hours at half the price. This guide compares the Anthropic, OpenAI, Gemini and Bedrock offerings, then covers what changes in your code: result matching, partial failures, caching, limits, data retention and when OpenAI's flex tier is the simpler option.",{"type":133,"level":134,"id":95,"text":96},"heading",2,{"type":126,"content":136},[137],"The test is simple: would anyone notice if the result arrived tomorrow morning instead of now? Anthropic's own list of use cases is large-scale evaluations, content moderation, data analysis and bulk content generation such as product descriptions. Typical candidates in the projects I work on:",{"type":139,"items":140},"list",[141,153,162,167],[142,146,147,152],{"tag":143,"children":144},"strong",[145],"Catalogue enrichment."," Descriptions, attribute extraction and translations for thousands of SKUs after a PIM import, see ",{"tag":148,"to":149,"children":150},"link","\u002Fblog\u002Fpimcore-erp-delta-sync",[151],"Pimcore ERP delta sync",".",[154,157,158,152],{"tag":143,"children":155},[156],"Eval runs."," Regression suites with hundreds of cases per prompt version, where a result in an hour is fine; see ",{"tag":148,"to":159,"children":160},"\u002Fblog\u002Fllm-evals-for-product-features",[161],"LLM evals for product features",[163,166],{"tag":143,"children":164},[165],"Backfills."," Classifying or summarising historical tickets, orders or documents after a schema or taxonomy change.",[168,171,172,176,177,180],{"tag":143,"children":169},[170],"Embeddings and moderation."," OpenAI batches cover ",{"tag":173,"children":174},"code",[175],"\u002Fv1\u002Fembeddings"," and ",{"tag":173,"children":178},[179],"\u002Fv1\u002Fmoderations","; Gemini has a dedicated batch embeddings call.",{"type":126,"content":182},[183],"What does not fit: anything a user waits for, and agent loops whose tools run in your code. Each batch request is one model turn, so an agent that calls a client tool needs a new request for every step. Server-side tools are different: on the Claude API web search, web fetch and code execution run inside the batch worker, which runs more loop iterations per turn than a synchronous call before pausing.",{"type":133,"level":134,"id":98,"text":99},{"type":186,"head":187,"rows":198},"table",[188,190,192,194,196],[189],"Aspect",[191],"Anthropic",[193],"OpenAI",[195],"Gemini",[197],"Bedrock",[199,209,220,231,242,260],[200,202,204,206,207],[201],"Price",[203],"50% of standard",[205],"50% less",[203],[208],"See Bedrock pricing",[210,212,214,216,218],[211],"Per batch",[213],"100,000 requests or 256 MB",[215],"50,000 requests, 200 MB file",[217],"20 MB inline, 2 GB file",[219],"Quotas per account",[221,223,225,227,229],[222],"Window",[224],"24 h, most under 1 h",[226],"24 h, fixed",[228],"24 h target",[230],"Asynchronous job",[232,234,236,238,240],[233],"Input",[235],"JSON request list",[237],"Uploaded JSONL file",[239],"Inline or JSONL file",[241],"JSONL in S3",[243,245,249,252,256],[244],"Match key",[246],{"tag":173,"children":247},[248],"custom_id",[250],{"tag":173,"children":251},[248],[253],{"tag":173,"children":254},[255],"key",[257],{"tag":173,"children":258},[259],"recordId",[261,263,265,267,269],[262],"Results kept",[264],"29 days",[266],"Output deleted after 30 days",[268],"6 weeks",[270],"Your S3 bucket",{"type":126,"content":272},[273,274,277,278,281,282,285],"A few constraints are easy to miss. An OpenAI input file can target only one model. Bedrock batch jobs that use the Converse format cannot use tool calling; you need the InvokeModel format with each model's native request body. And on the Claude API, ",{"tag":173,"children":275},[276],"stream",", the fast-mode ",{"tag":173,"children":279},[280],"speed"," parameter and ",{"tag":173,"children":283},[284],"max_tokens: 0"," are rejected inside a batch; almost everything else, including vision, tool use and most beta features, works.",{"type":133,"level":134,"id":101,"text":102},{"type":288,"attrs":289,"inner":293,"caption":294},"diagram",{"viewBox":290,"role":291,"aria-labelledby":292},"0 0 720 170","img","lba1-t lba1-d","\u003Ctitle id=\"lba1-t\">Batch lifecycle from source records to stored results\u003C\u002Ftitle>\u003Cdesc id=\"lba1-d\">Source records become requests with deterministic ids, the batch is submitted and polled until it ends, results are joined by id, and failures go into a new batch.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"lba1-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"20\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"79\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Build\u003C\u002Ftext>\u003Ctext x=\"79\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">id per record\u003C\u002Ftext>\u003Cpath d=\"M138 85H158\" class=\"d-line\" marker-end=\"url(#lba1-ah)\" \u002F>\u003Crect x=\"160\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"220\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Submit\u003C\u002Ftext>\u003Ctext x=\"220\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">one model\u003C\u002Ftext>\u003Cpath d=\"M279 85H299\" class=\"d-line\" marker-end=\"url(#lba1-ah)\" \u002F>\u003Crect x=\"301\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"360\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Poll\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">until ended\u003C\u002Ftext>\u003Cpath d=\"M419 85H439\" class=\"d-line\" marker-end=\"url(#lba1-ah)\" \u002F>\u003Crect x=\"441\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"500\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Join\u003C\u002Ftext>\u003Ctext x=\"500\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">by custom_id\u003C\u002Ftext>\u003Cpath d=\"M560 85H580\" class=\"d-line\" marker-end=\"url(#lba1-ah)\" \u002F>\u003Crect x=\"582\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"641\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Retry\u003C\u002Ftext>\u003Ctext x=\"641\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">failed only\u003C\u002Ftext>",[295],"The loop closes on failures: errored and expired requests go into the next batch, not back into the synchronous API.",{"type":126,"content":297},[298,299,302,303,306,307,310,311,314,315,318,319,314,321,314,324,314,327,314,330,176,333,336,337,340],"On the Claude API a new batch starts with ",{"tag":173,"children":300},[301],"processing_status"," set to ",{"tag":173,"children":304},[305],"in_progress"," and moves to ",{"tag":173,"children":308},[309],"ended"," when every request has finished; results can be downloaded once all messages are complete or after 24 hours, whichever comes first. OpenAI has more states: ",{"tag":173,"children":312},[313],"validating",", ",{"tag":173,"children":316},[317],"failed"," if the input file is invalid, ",{"tag":173,"children":320},[305],{"tag":173,"children":322},[323],"finalizing",{"tag":173,"children":325},[326],"completed",{"tag":173,"children":328},[329],"expired",{"tag":173,"children":331},[332],"cancelling",{"tag":173,"children":334},[335],"cancelled",". Gemini uses ",{"tag":173,"children":338},[339],"JOB_STATE_*"," values from pending to expired. Bedrock can notify you through EventBridge instead of polling.",{"type":133,"level":134,"id":104,"text":105},{"type":126,"content":343},[344,345,347,348,351],"Both Anthropic and OpenAI state that results may not come back in input order, and tell you to match on ",{"tag":173,"children":346},[248],". Make that id deterministic: derive it from the source record and the prompt version, so a rerun produces the same id and your writer can upsert instead of duplicating. On the Claude API it must match ",{"tag":173,"children":349},[350],"^[a-zA-Z0-9_-]{1,64}$",", which rules out raw SKUs with slashes or dots.",{"type":173,"code":353},"import hashlib, time, anthropic\n\nclient = anthropic.Anthropic()\nPROMPT_VERSION = \"desc-v7\"\n\ndef cid(sku: str) -> str:   # deterministic, safe characters, \u003C= 64 chars\n    return f\"{PROMPT_VERSION}-{hashlib.sha1(sku.encode()).hexdigest()[:20]}\"\n\nrequests = [{\n    \"custom_id\": cid(p[\"sku\"]),\n    \"params\": {\n        \"model\": \"claude-sonnet-5-5\",\n        \"max_tokens\": 800,\n        \"system\": [{\"type\": \"text\", \"text\": STYLE_GUIDE,             # identical in every request\n                    \"cache_control\": {\"type\": \"ephemeral\", \"ttl\": \"1h\"}}],\n        \"messages\": [{\"role\": \"user\", \"content\": render_product(p)}],\n    },\n} for p in products]\n\nbatch = client.messages.batches.create(requests=requests)\nwhile client.messages.batches.retrieve(batch.id).processing_status != \"ended\":\n    time.sleep(60)\n\nretry = []\nfor r in client.messages.batches.results(batch.id):\n    if r.result.type == \"succeeded\":\n        upsert_description(r.custom_id, r.result.message)   # idempotent write\n    else:                                                    # errored, canceled, expired\n        retry.append(r.custom_id)",{"type":126,"content":355},[356,357,360],"Validation of each request's ",{"tag":173,"children":358},[359],"params"," happens asynchronously on the Claude API, so a typo in one field only surfaces when the whole batch has ended. Anthropic's advice is to dry-run one request shape against the synchronous Messages API first. I do that in the same script: send the first request synchronously, check the response against the schema, then submit the rest.",{"type":133,"level":134,"id":107,"text":108},{"type":126,"content":363},[364,365,314,368,314,371,374,375,377,378,152],"A batch never simply succeeds or fails. On the Claude API every request ends as ",{"tag":173,"children":366},[367],"succeeded",{"tag":173,"children":369},[370],"errored",{"tag":173,"children":372},[373],"canceled"," or ",{"tag":173,"children":376},[329],", and only the first is billed. Expiry is not hypothetical: Anthropic notes that processing may slow down with demand and your request volume, in which case more requests expire after 24 hours. OpenAI writes successes to an output file and failures to an error file; when a batch expires, completed requests stay available and are billed, and the rest appear with the code ",{"tag":173,"children":379},[380],"batch_expired",{"type":139,"items":382},[383,388,393,406],[384,387],{"tag":143,"children":385},[386],"Separate the error classes."," Validation errors need a code fix; server errors and expiries go into the next batch unchanged.",[389,392],{"tag":143,"children":390},[391],"Cap the retry rounds."," Two or three resubmissions, then park the records for a human look.",[394,397,398,401,402,152],{"tag":143,"children":395},[396],"Validate content, not only status."," A succeeded request can still hit ",{"tag":173,"children":399},[400],"max_tokens"," or return JSON that fails your schema, see ",{"tag":148,"to":403,"children":404},"\u002Fblog\u002Fstructured-outputs-json-schema-llm",[405],"structured outputs",[407,410],{"tag":143,"children":408},[409],"Split very large jobs."," Anthropic recommends breaking large datasets into multiple batches; it also makes progress and cost visible per chunk.",{"type":133,"level":134,"id":110,"text":111},{"type":126,"content":413},[414,415,418],"Batch jobs are usually the same long instructions applied to many records, which is the ideal shape for prompt caching. Anthropic states that caching and batch discounts stack. The catch is that batch requests run concurrently and in any order, so cache hits are best effort; Anthropic reports typical hit rates between 30% and 98% depending on traffic patterns. Its advice is to include identical ",{"tag":173,"children":416},[417],"cache_control"," blocks in every request and, because a batch can run longer than the five-minute cache lifetime, to consider the one-hour cache. Gemini also supports context caching for batch requests at its standard caching rates.",{"type":126,"content":420},[421,422,152],"Do the arithmetic before relying on it. The one-hour cache write costs twice the base input price instead of 1.25 times, so it pays off only when the shared prefix is long and the batch is large enough to read it many times. For the caching mechanics themselves, see ",{"tag":148,"to":423,"children":424},"\u002Fblog\u002Fllm-cost-latency-prompt-caching-routing",[425],"prompt caching and model routing",{"type":133,"level":134,"id":113,"text":114},{"type":126,"content":428},[429],"Batch traffic has its own limits on both platforms, which is half the point. Anthropic says Batches API usage does not affect Messages API rate limits; instead it limits HTTP requests to the batch endpoints and the number of batch requests waiting in the processing queue, shared across models: 200,000 queued requests on the Start tier, 300,000 on Build and 500,000 on Scale, with at most 100,000 per batch. OpenAI's batch limits are a separate pool calculated from queued input tokens per model, and you can create up to 2,000 batches per hour.",{"type":431,"variant":432,"title":433,"body":434},"callout","warn","Batches can overshoot a spend limit",[435],[436],"Anthropic notes that, because of high throughput and concurrent processing, batches may go slightly over a workspace's configured spend limit. If a workspace limit is your only guard against a runaway backfill, add your own check: estimate tokens per request, multiply by the batch size and refuse to submit above a budget.",{"type":126,"content":438},[439,440,443,444,446],"For long outputs there is one batch-only feature on the Claude API: the ",{"tag":173,"children":441},[442],"output-300k-2026-03-24"," beta header raises ",{"tag":173,"children":445},[400]," to 300,000 on the current Opus, Sonnet and Haiku 5.5 generation and several earlier models, for book-length drafts or exhaustive extraction. A single generation of that size can take over an hour, so it eats into the 24-hour window.",{"type":133,"level":134,"id":116,"text":117},{"type":126,"content":449},[450,451,454,455,458,459,462],"OpenAI offers a middle ground. Setting ",{"tag":173,"children":452},[453],"service_tier"," to ",{"tag":173,"children":456},[457],"flex"," on a Responses or Chat Completions call gives batch-rate pricing on a normal synchronous request, in exchange for slower responses and occasional unavailability. When capacity is missing you get a 429 you are not charged for, and you either back off and retry or resend with ",{"tag":173,"children":460},[461],"service_tier: auto"," at standard price. The SDK default timeout is ten minutes, and OpenAI's examples raise it to fifteen.",{"type":186,"head":464,"rows":470},[465,467,468],[466],"Question",[9],[469],"Flex processing",[471,478,483,490],[472,474,476],[473],"Code shape",[475],"Submit file, poll, parse results",[477],"Ordinary request and response",[479,480,482],[201],[481],"Batch rates",[481],[484,486,488],[485],"Failure mode",[487],"Expired or errored rows in the results",[489],"429 resource unavailable, not billed",[491,493,495],[492],"Best for",[494],"Tens of thousands of independent requests",[496],"Low-priority work inside existing code paths",{"type":133,"level":134,"id":119,"text":120},{"type":126,"content":499},[500,501,504],"Asynchronous processing means the provider stores your inputs and outputs until you fetch them. Anthropic keeps batch request and response data for up to 29 days after creation, lets you delete a finished batch through ",{"tag":173,"children":502},[503],"DELETE \u002Fv1\u002Fmessages\u002Fbatches\u002F{batch_id}",", isolates batches per workspace and allows result downloads in the Console to be disabled; zero data retention eligibility is listed per feature in its data retention page. OpenAI deletes output files 30 days after the batch completes, Gemini keeps results for 6 weeks, and Bedrock writes them to an S3 bucket you control.",{"type":126,"content":506},[507,508,512,513,517],"For personal data that means three things: delete batches once results are stored, keep the provider and region in your records of processing, and check whether your contract's retention promises cover batch storage at all. The broader picture is in ",{"tag":148,"to":509,"children":510},"\u002Fblog\u002Fgdpr-llm-api-eu-data-residency",[511],"GDPR LLM data residency","; if a backfill touches customer records, run it through ",{"tag":148,"to":514,"children":515},"\u002Fblog\u002Fpii-redaction-llm-pipelines",[516],"PII redaction"," first.",{"type":133,"level":134,"id":122,"text":123},{"type":139,"ordered":520,"items":521},true,[522,526,529,532,535,538,541,544],[523],{"tag":524,"href":38,"children":525},"a",[37],[527],{"tag":524,"href":41,"children":528},[40],[530],{"tag":524,"href":44,"children":531},[43],[533],{"tag":524,"href":47,"children":534},[46],[536],{"tag":524,"href":50,"children":537},[49],[539],{"tag":524,"href":53,"children":540},[52],[542],{"tag":524,"href":56,"children":543},[55],[545],{"tag":524,"href":59,"children":546},[58],[548,614,672,725],{"slug":549,"published":5,"minutes":550,"category":7,"tags":551,"keywords":557,"about":569,"sources":579,"cover":608,"og":609,"expertise":62,"locales":610,"lang":64,"title":611,"description":612,"coverAlt":613},"llm-rate-limits-retries-provider-fallback",10,[552,553,554,555,556],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[558,559,560,561,562,563,564,565,566,567,568],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[570,573,576],{"name":571,"url":572},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":574,"url":575},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":577,"url":578},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[580,582,585,587,590,593,596,599,602,605],{"title":581,"url":41},"Claude Platform Docs: Rate limits",{"title":583,"url":584},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":586,"url":53},"OpenAI API Docs: Rate limits",{"title":588,"url":589},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":591,"url":592},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":594,"url":595},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":597,"url":598},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":600,"url":601},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":603,"url":604},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":606,"url":607},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[64,65,66],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":615,"published":5,"minutes":616,"category":7,"tags":617,"keywords":622,"about":633,"sources":643,"cover":666,"og":667,"expertise":62,"locales":668,"lang":64,"title":669,"description":670,"coverAlt":671},"prompt-versioning-management",8,[618,619,11,620,621],"Prompt management","Versioning","Evals","Langfuse",[623,624,625,626,627,628,629,630,631,632],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[634,637,640],{"name":635,"url":636},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":638,"url":639},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":641,"url":642},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[644,647,650,653,656,659,662,665],{"title":645,"url":646},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":648,"url":649},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":651,"url":652},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":654,"url":655},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":657,"url":658},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":660,"url":661},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":663,"url":664},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":43,"url":44},"\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[64,65,66],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":673,"published":674,"minutes":6,"category":7,"tags":675,"keywords":681,"about":693,"sources":700,"cover":719,"og":720,"expertise":62,"locales":721,"lang":64,"title":722,"description":723,"coverAlt":724},"structured-outputs-json-schema-llm","2026-10-16",[676,677,678,679,680],"Structured outputs","JSON Schema","Constrained decoding","LLM APIs","Validation",[682,683,684,685,686,687,688,689,690,691,692],"structured outputs LLM","JSON schema LLM output","OpenAI structured outputs strict mode","Claude structured outputs","Gemini structured output","constrained decoding","JSON mode vs structured outputs","validate LLM JSON output","Zod structured output","Pydantic LLM output","vLLM structured outputs",[694,697,699],{"name":695,"url":696},"JSON","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FJSON",{"name":677,"url":698},"https:\u002F\u002Fjson-schema.org\u002F",{"name":33,"url":34},[701,704,707,710,713,716],{"title":702,"url":703},"OpenAI API docs: Structured model outputs","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fstructured-outputs",{"title":705,"url":706},"Claude API docs: Structured outputs","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fstructured-outputs",{"title":708,"url":709},"Gemini API docs: Structured outputs","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fstructured-output",{"title":711,"url":712},"vLLM docs: Structured Outputs","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fstructured_outputs.html",{"title":714,"url":715},"Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.02442",{"title":717,"url":718},"Geng et al.: JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.10868","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fog.jpg",[64,65,66],"Structured outputs with JSON Schema: what the guarantee covers, and what it does not","How structured outputs and constrained decoding work on OpenAI, Claude, Gemini and vLLM, which schema rules matter, and what you still validate.","A fixed four-compartment frame, three blocks seated, one too-wide block hovering above.",{"slug":726,"published":674,"minutes":6,"category":7,"tags":727,"keywords":733,"about":746,"sources":754,"cover":776,"og":777,"expertise":62,"locales":778,"lang":64,"title":779,"description":780,"coverAlt":781},"evaluating-ai-agents-trajectories",[728,729,730,731,732],"Agent evals","Trajectories","Tool calling","pass^k","LLM-as-judge",[734,735,736,737,738,739,740,741,742,743,744,745],"evaluating AI agents","agent trajectory evaluation","agent evals","tool call accuracy","trajectory match evaluator","pass^k metric","tau-bench","LLM as a judge for agents","agent regression testing","outcome vs trajectory evaluation","agentevals","tool correctness metric",[747,750,753],{"name":748,"url":749},"Intelligent agent","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIntelligent_agent",{"name":751,"url":752},"Software testing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSoftware_testing",{"name":33,"url":34},[755,758,761,764,767,770,773],{"title":756,"url":757},"Anthropic: Demystifying evals for AI agents (Jan 2026)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents",{"title":759,"url":760},"Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045",{"title":762,"url":763},"Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07982",{"title":765,"url":766},"LangChain: agentevals (GitHub)","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals",{"title":768,"url":769},"DeepEval docs: Tool Correctness","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fmetrics-tool-correctness",{"title":771,"url":772},"Ragas docs: Agentic or tool use metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002Fagents\u002F",{"title":774,"url":775},"OpenAI API docs: Trace grading","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ftrace-grading","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fog.jpg",[64,65,66],"Evaluating AI agents: grading trajectories, outcomes and reliability","How to evaluate AI agents: outcome vs trajectory graders, trajectory match modes, pass^k, isolated environments and LLM judges that hold up.","Two dotted paths of circles between two squares, a wide bracket framing the path that bends.",1792222776049]