> When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching. > > Web page: https://balazscsorba.com/blog/llm-batch-api-offline-workloads · Language: English · Also available in: [Deutsch](https://balazscsorba.com/de/blog/llm-batch-api-offline-workloads.md) · [Magyar](https://balazscsorba.com/hu/blog/llm-batch-api-offline-workloads.md) > Author: Balázs Csorba · Published: 2026-10-17 · Keywords: llm batch api, anthropic message batches api, openai batch api, gemini batch mode, batch inference llm cost, batch api 50 percent discount, openai flex processing, bedrock batch inference, llm offline processing, batch api prompt caching [Blog](https://balazscsorba.com/blog)/LLMOps & evals # LLM batch APIs: half-price inference for work that can wait When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching. [Balázs Csorba](https://balazscsorba.com/about)·October 17, 2026·9 min read - Batch API - Cost - LLMOps - Prompt caching - Data pipelines ![A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.](https://balazscsorba.com/images/blog/llm-batch-api-offline-workloads/cover.webp?v=417281c599) ## Key takeaways - Anthropic, OpenAI and Gemini all charge 50% of standard prices for batch requests, in exchange for asynchronous results within a 24-hour window. - Results come back in any order: always join them to source records by custom\_id or key, never by position. - Batch limits are a separate pool from synchronous limits, so moving backfills and evals to batch also protects your interactive traffic. - Batch and prompt caching discounts stack, but batch cache hits are best effort; use identical cache\_control blocks and consider the 1-hour cache. - Batch data sits on provider storage for weeks: Anthropic keeps results 29 days, OpenAI deletes output files after 30 days, Gemini after 6 weeks. On this page 1. [Which workloads belong in a batch](https://balazscsorba.com/#when-batch-fits) 2. [Anthropic, OpenAI, Gemini and Bedrock compared](https://balazscsorba.com/#provider-comparison) 3. [The batch lifecycle](https://balazscsorba.com/#batch-lifecycle) 4. [Building requests you can match and retry](https://balazscsorba.com/#building-requests) 5. [Partial failures and what you pay for](https://balazscsorba.com/#results-and-failures) 6. [Stacking batch and prompt caching discounts](https://balazscsorba.com/#caching-with-batches) 7. [Limits, throughput and spend](https://balazscsorba.com/#limits-throughput) 8. [When flex processing is the simpler choice](https://balazscsorba.com/#flex-alternative) 9. [Data retention and GDPR](https://balazscsorba.com/#data-retention) 10. [Sources](https://balazscsorba.com/#sources) Listen to this article 0:000:00 A surprising share of LLM spend never touches a user. Re-classifying a support archive after a taxonomy change, writing product descriptions for a catalogue import, running a nightly eval suite, re-embedding documents after a model switch: none of these needs an answer in two seconds, yet many teams push them through the same synchronous endpoint as their chat feature, at full price and competing for the same rate limits. Batch APIs exist for exactly this work. You submit many requests at once, the provider processes them when it has capacity, and you collect a file of results within 24 hours at half the price. This guide compares the Anthropic, OpenAI, Gemini and Bedrock offerings, then covers what changes in your code: result matching, partial failures, caching, limits, data retention and when OpenAI's flex tier is the simpler option. ## Which workloads belong in a batch The test is simple: would anyone notice if the result arrived tomorrow morning instead of now? Anthropic's own list of use cases is large-scale evaluations, content moderation, data analysis and bulk content generation such as product descriptions. Typical candidates in the projects I work on: - **Catalogue enrichment.** Descriptions, attribute extraction and translations for thousands of SKUs after a PIM import, see [Pimcore ERP delta sync](https://balazscsorba.com/blog/pimcore-erp-delta-sync). - **Eval runs.** Regression suites with hundreds of cases per prompt version, where a result in an hour is fine; see [LLM evals for product features](https://balazscsorba.com/blog/llm-evals-for-product-features). - **Backfills.** Classifying or summarising historical tickets, orders or documents after a schema or taxonomy change. - **Embeddings and moderation.** OpenAI batches cover `/v1/embeddings` and `/v1/moderations`; Gemini has a dedicated batch embeddings call. What does not fit: anything a user waits for, and agent loops whose tools run in your code. Each batch request is one model turn, so an agent that calls a client tool needs a new request for every step. Server-side tools are different: on the Claude API web search, web fetch and code execution run inside the batch worker, which runs more loop iterations per turn than a synchronous call before pausing. ## Anthropic, OpenAI, Gemini and Bedrock compared Aspect Anthropic OpenAI Gemini Bedrock Price 50% of standard 50% less 50% of standard See Bedrock pricing Per batch 100,000 requests or 256 MB 50,000 requests, 200 MB file 20 MB inline, 2 GB file Quotas per account Window 24 h, most under 1 h 24 h, fixed 24 h target Asynchronous job Input JSON request list Uploaded JSONL file Inline or JSONL file JSONL in S3 Match key `custom_id` `custom_id` `key` `recordId` Results kept 29 days Output deleted after 30 days 6 weeks Your S3 bucket A few constraints are easy to miss. An OpenAI input file can target only one model. Bedrock batch jobs that use the Converse format cannot use tool calling; you need the InvokeModel format with each model's native request body. And on the Claude API, `stream`, the fast-mode `speed` parameter and `max_tokens: 0` are rejected inside a batch; almost everything else, including vision, tool use and most beta features, works. ## The batch lifecycle The loop closes on failures: errored and expired requests go into the next batch, not back into the synchronous API. On the Claude API a new batch starts with `processing_status` set to `in_progress` and moves to `ended` when every request has finished; results can be downloaded once all messages are complete or after 24 hours, whichever comes first. OpenAI has more states: `validating`, `failed` if the input file is invalid, `in_progress`, `finalizing`, `completed`, `expired`, `cancelling` and `cancelled`. Gemini uses `JOB_STATE_*` values from pending to expired. Bedrock can notify you through EventBridge instead of polling. ## Building requests you can match and retry Both Anthropic and OpenAI state that results may not come back in input order, and tell you to match on `custom_id`. Make that id deterministic: derive it from the source record and the prompt version, so a rerun produces the same id and your writer can upsert instead of duplicating. On the Claude API it must match `^[a-zA-Z0-9_-]{1,64}$`, which rules out raw SKUs with slashes or dots. ``` import hashlib, time, anthropic client = anthropic.Anthropic() PROMPT_VERSION = "desc-v7" def cid(sku: str) -> str: # deterministic, safe characters, <= 64 chars return f"{PROMPT_VERSION}-{hashlib.sha1(sku.encode()).hexdigest()[:20]}" requests = [{ "custom_id": cid(p["sku"]), "params": { "model": "claude-sonnet-5-5", "max_tokens": 800, "system": [{"type": "text", "text": STYLE_GUIDE, # identical in every request "cache_control": {"type": "ephemeral", "ttl": "1h"}}], "messages": [{"role": "user", "content": render_product(p)}], }, } for p in products] batch = client.messages.batches.create(requests=requests) while client.messages.batches.retrieve(batch.id).processing_status != "ended": time.sleep(60) retry = [] for r in client.messages.batches.results(batch.id): if r.result.type == "succeeded": upsert_description(r.custom_id, r.result.message) # idempotent write else: # errored, canceled, expired retry.append(r.custom_id) ``` Validation of each request's `params` happens asynchronously on the Claude API, so a typo in one field only surfaces when the whole batch has ended. Anthropic's advice is to dry-run one request shape against the synchronous Messages API first. I do that in the same script: send the first request synchronously, check the response against the schema, then submit the rest. ## Partial failures and what you pay for A batch never simply succeeds or fails. On the Claude API every request ends as `succeeded`, `errored`, `canceled` or `expired`, and only the first is billed. Expiry is not hypothetical: Anthropic notes that processing may slow down with demand and your request volume, in which case more requests expire after 24 hours. OpenAI writes successes to an output file and failures to an error file; when a batch expires, completed requests stay available and are billed, and the rest appear with the code `batch_expired`. - **Separate the error classes.** Validation errors need a code fix; server errors and expiries go into the next batch unchanged. - **Cap the retry rounds.** Two or three resubmissions, then park the records for a human look. - **Validate content, not only status.** A succeeded request can still hit `max_tokens` or return JSON that fails your schema, see [structured outputs](https://balazscsorba.com/blog/structured-outputs-json-schema-llm). - **Split very large jobs.** Anthropic recommends breaking large datasets into multiple batches; it also makes progress and cost visible per chunk. ## Stacking batch and prompt caching discounts Batch jobs are usually the same long instructions applied to many records, which is the ideal shape for prompt caching. Anthropic states that caching and batch discounts stack. The catch is that batch requests run concurrently and in any order, so cache hits are best effort; Anthropic reports typical hit rates between 30% and 98% depending on traffic patterns. Its advice is to include identical `cache_control` blocks in every request and, because a batch can run longer than the five-minute cache lifetime, to consider the one-hour cache. Gemini also supports context caching for batch requests at its standard caching rates. Do the arithmetic before relying on it. The one-hour cache write costs twice the base input price instead of 1.25 times, so it pays off only when the shared prefix is long and the batch is large enough to read it many times. For the caching mechanics themselves, see [prompt caching and model routing](https://balazscsorba.com/blog/llm-cost-latency-prompt-caching-routing). ## Limits, throughput and spend Batch traffic has its own limits on both platforms, which is half the point. Anthropic says Batches API usage does not affect Messages API rate limits; instead it limits HTTP requests to the batch endpoints and the number of batch requests waiting in the processing queue, shared across models: 200,000 queued requests on the Start tier, 300,000 on Build and 500,000 on Scale, with at most 100,000 per batch. OpenAI's batch limits are a separate pool calculated from queued input tokens per model, and you can create up to 2,000 batches per hour. **Batches can overshoot a spend limit** Anthropic notes that, because of high throughput and concurrent processing, batches may go slightly over a workspace's configured spend limit. If a workspace limit is your only guard against a runaway backfill, add your own check: estimate tokens per request, multiply by the batch size and refuse to submit above a budget. For long outputs there is one batch-only feature on the Claude API: the `output-300k-2026-03-24` beta header raises `max_tokens` to 300,000 on the current Opus, Sonnet and Haiku 5.5 generation and several earlier models, for book-length drafts or exhaustive extraction. A single generation of that size can take over an hour, so it eats into the 24-hour window. ## When flex processing is the simpler choice OpenAI offers a middle ground. Setting `service_tier` to `flex` on a Responses or Chat Completions call gives batch-rate pricing on a normal synchronous request, in exchange for slower responses and occasional unavailability. When capacity is missing you get a 429 you are not charged for, and you either back off and retry or resend with `service_tier: auto` at standard price. The SDK default timeout is ten minutes, and OpenAI's examples raise it to fifteen. Question Batch API Flex processing Code shape Submit file, poll, parse results Ordinary request and response Price Batch rates Batch rates Failure mode Expired or errored rows in the results 429 resource unavailable, not billed Best for Tens of thousands of independent requests Low-priority work inside existing code paths ## Data retention and GDPR Asynchronous processing means the provider stores your inputs and outputs until you fetch them. Anthropic keeps batch request and response data for up to 29 days after creation, lets you delete a finished batch through `DELETE /v1/messages/batches/{batch_id}`, isolates batches per workspace and allows result downloads in the Console to be disabled; zero data retention eligibility is listed per feature in its data retention page. OpenAI deletes output files 30 days after the batch completes, Gemini keeps results for 6 weeks, and Bedrock writes them to an S3 bucket you control. For personal data that means three things: delete batches once results are stored, keep the provider and region in your records of processing, and check whether your contract's retention promises cover batch storage at all. The broader picture is in [GDPR LLM data residency](https://balazscsorba.com/blog/gdpr-llm-api-eu-data-residency); if a backfill touches customer records, run it through [PII redaction](https://balazscsorba.com/blog/pii-redaction-llm-pipelines) first. ## Sources 1. [Claude Platform Docs: Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing) 2. [Claude Platform Docs: Rate limits (Message Batches API)](https://platform.claude.com/docs/en/api/rate-limits) 3. [Claude Platform Docs: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) 4. [OpenAI API Docs: Batch API](https://developers.openai.com/api/docs/guides/batch) 5. [OpenAI API Docs: Flex processing](https://developers.openai.com/api/docs/guides/flex-processing) 6. [OpenAI API Docs: Rate limits (batch queue limits)](https://developers.openai.com/api/docs/guides/rate-limits) 7. [Google AI for Developers: Gemini Batch API](https://ai.google.dev/gemini-api/docs/batch-api) 8. [Amazon Bedrock User Guide: Batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) ## Frequently asked questions How much cheaper is a batch API? Anthropic, OpenAI and Google Gemini all price batch requests at 50% of standard rates. On the Claude API that applies to input, output and special tokens, so Claude Opus 5.5 costs 2 dollars per million input tokens and 10 dollars per million output tokens in a batch. Prompt caching discounts apply on top. How long does a batch take? All three providers work with a 24-hour window. Anthropic says most batches finish in less than an hour, but requests that are not processed within 24 hours expire. Plan pipelines around the 24-hour worst case, not the typical case. Can I use tools and structured output in batch requests? On the Claude API, almost any Messages request can be batched, including tool use, server tools, vision, system prompts and multi-turn conversations; stream, speed and max\_tokens 0 are not supported. Client tools still need a round trip, so each tool call means a new request. On Bedrock, tool calling needs the InvokeModel format rather than Converse. What happens to failed requests in a batch? Each request gets its own result. On the Claude API, errored, canceled and expired requests are not billed. On OpenAI, failures go to a separate error file, and when a batch expires you pay for the requests that did complete. Collect the failures and resubmit them in a new batch. Batch API or OpenAI flex processing? Flex is a synchronous request with service\_tier set to flex, priced at batch rates but slower and sometimes unavailable, in which case it returns a 429 you are not charged for. It suits jobs that want simple request-response code. A batch suits large volumes where you can wait for a file of results. Written by Balázs Csorba Senior fullstack & AI engineer in Styria, Austria – 10+ years of Vue, Nuxt, Node.js and PHP, now building tooling for AI agents. [AI engineering & MCP servers →](https://balazscsorba.com/expertise/ai-engineer)[About me →](https://balazscsorba.com/about) ## More articles - [LLM rate limits in production: retries, backoff and provider fallback](https://balazscsorba.com/blog/llm-rate-limits-retries-provider-fallback) - [Prompt versioning in production: git, registries, labels and rollbacks](https://balazscsorba.com/blog/prompt-versioning-management) - [Structured outputs with JSON Schema: what the guarantee covers, and what it does not](https://balazscsorba.com/blog/structured-outputs-json-schema-llm) - [Evaluating AI agents: grading trajectories, outcomes and reliability](https://balazscsorba.com/blog/evaluating-ai-agents-trajectories) ## Sounds like what you need? Tell me about your project or role – I’d love to hear from you. [Get in touch](mailto:contact@balazscsorba.com) [Connect on LinkedIn](https://www.linkedin.com/in/balazs-csorba)