LLM batch APIs: half-price inference for work that can wait

A surprising share of LLM spend never touches a user. Re-classifying a support archive after a taxonomy change, writing product descriptions for a catalogue import, running a nightly eval suite, re-embedding documents after a model switch: none of these needs an answer in two seconds, yet many teams push them through the same synchronous endpoint as their chat feature, at full price and competing for the same rate limits.

Batch APIs exist for exactly this work. You submit many requests at once, the provider processes them when it has capacity, and you collect a file of results within 24 hours at half the price. This guide compares the Anthropic, OpenAI, Gemini and Bedrock offerings, then covers what changes in your code: result matching, partial failures, caching, limits, data retention and when OpenAI's flex tier is the simpler option.

Which workloads belong in a batch

The test is simple: would anyone notice if the result arrived tomorrow morning instead of now? Anthropic's own list of use cases is large-scale evaluations, content moderation, data analysis and bulk content generation such as product descriptions. Typical candidates in the projects I work on:

• Catalogue enrichment. Descriptions, attribute extraction and translations for thousands of SKUs after a PIM import, see Pimcore ERP delta sync.
• Eval runs. Regression suites with hundreds of cases per prompt version, where a result in an hour is fine; see LLM evals for product features.
• Backfills. Classifying or summarising historical tickets, orders or documents after a schema or taxonomy change.
• Embeddings and moderation. OpenAI batches cover /v1/embeddings and /v1/moderations; Gemini has a dedicated batch embeddings call.

What does not fit: anything a user waits for, and agent loops whose tools run in your code. Each batch request is one model turn, so an agent that calls a client tool needs a new request for every step. Server-side tools are different: on the Claude API web search, web fetch and code execution run inside the batch worker, which runs more loop iterations per turn than a synchronous call before pausing.

Anthropic, OpenAI, Gemini and Bedrock compared

A few constraints are easy to miss. An OpenAI input file can target only one model. Bedrock batch jobs that use the Converse format cannot use tool calling; you need the InvokeModel format with each model's native request body. And on the Claude API, stream, the fast-mode speed parameter and max_tokens: 0 are rejected inside a batch; almost everything else, including vision, tool use and most beta features, works.

The batch lifecycle

On the Claude API a new batch starts with processing_status set to in_progress and moves to ended when every request has finished; results can be downloaded once all messages are complete or after 24 hours, whichever comes first. OpenAI has more states: validating, failed if the input file is invalid, in_progress, finalizing, completed, expired, cancelling and cancelled. Gemini uses JOB_STATE_* values from pending to expired. Bedrock can notify you through EventBridge instead of polling.

Building requests you can match and retry

Both Anthropic and OpenAI state that results may not come back in input order, and tell you to match on custom_id. Make that id deterministic: derive it from the source record and the prompt version, so a rerun produces the same id and your writer can upsert instead of duplicating. On the Claude API it must match ^[a-zA-Z0-9_-]{1,64}$, which rules out raw SKUs with slashes or dots.

import hashlib, time, anthropic

client = anthropic.Anthropic()
PROMPT_VERSION = "desc-v7"

def cid(sku: str) -> str:   # deterministic, safe characters, <= 64 chars
    return f"{PROMPT_VERSION}-{hashlib.sha1(sku.encode()).hexdigest()[:20]}"

requests = [{
    "custom_id": cid(p["sku"]),
    "params": {
        "model": "claude-sonnet-5-5",
        "max_tokens": 800,
        "system": [{"type": "text", "text": STYLE_GUIDE,             # identical in every request
                    "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
        "messages": [{"role": "user", "content": render_product(p)}],
    },
} for p in products]

batch = client.messages.batches.create(requests=requests)
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)

retry = []
for r in client.messages.batches.results(batch.id):
    if r.result.type == "succeeded":
        upsert_description(r.custom_id, r.result.message)   # idempotent write
    else:                                                    # errored, canceled, expired
        retry.append(r.custom_id)

Validation of each request's params happens asynchronously on the Claude API, so a typo in one field only surfaces when the whole batch has ended. Anthropic's advice is to dry-run one request shape against the synchronous Messages API first. I do that in the same script: send the first request synchronously, check the response against the schema, then submit the rest.

Partial failures and what you pay for

A batch never simply succeeds or fails. On the Claude API every request ends as succeeded, errored, canceled or expired, and only the first is billed. Expiry is not hypothetical: Anthropic notes that processing may slow down with demand and your request volume, in which case more requests expire after 24 hours. OpenAI writes successes to an output file and failures to an error file; when a batch expires, completed requests stay available and are billed, and the rest appear with the code batch_expired.

• Separate the error classes. Validation errors need a code fix; server errors and expiries go into the next batch unchanged.
• Cap the retry rounds. Two or three resubmissions, then park the records for a human look.
• Validate content, not only status. A succeeded request can still hit max_tokens or return JSON that fails your schema, see structured outputs.
• Split very large jobs. Anthropic recommends breaking large datasets into multiple batches; it also makes progress and cost visible per chunk.

Stacking batch and prompt caching discounts

Batch jobs are usually the same long instructions applied to many records, which is the ideal shape for prompt caching. Anthropic states that caching and batch discounts stack. The catch is that batch requests run concurrently and in any order, so cache hits are best effort; Anthropic reports typical hit rates between 30% and 98% depending on traffic patterns. Its advice is to include identical cache_control blocks in every request and, because a batch can run longer than the five-minute cache lifetime, to consider the one-hour cache. Gemini also supports context caching for batch requests at its standard caching rates.

Do the arithmetic before relying on it. The one-hour cache write costs twice the base input price instead of 1.25 times, so it pays off only when the shared prefix is long and the batch is large enough to read it many times. For the caching mechanics themselves, see prompt caching and model routing.

Limits, throughput and spend

Batch traffic has its own limits on both platforms, which is half the point. Anthropic says Batches API usage does not affect Messages API rate limits; instead it limits HTTP requests to the batch endpoints and the number of batch requests waiting in the processing queue, shared across models: 200,000 queued requests on the Start tier, 300,000 on Build and 500,000 on Scale, with at most 100,000 per batch. OpenAI's batch limits are a separate pool calculated from queued input tokens per model, and you can create up to 2,000 batches per hour.

For long outputs there is one batch-only feature on the Claude API: the output-300k-2026-03-24 beta header raises max_tokens to 300,000 on the current Opus, Sonnet and Haiku 5.5 generation and several earlier models, for book-length drafts or exhaustive extraction. A single generation of that size can take over an hour, so it eats into the 24-hour window.

When flex processing is the simpler choice

OpenAI offers a middle ground. Setting service_tier to flex on a Responses or Chat Completions call gives batch-rate pricing on a normal synchronous request, in exchange for slower responses and occasional unavailability. When capacity is missing you get a 429 you are not charged for, and you either back off and retry or resend with service_tier: auto at standard price. The SDK default timeout is ten minutes, and OpenAI's examples raise it to fifteen.

Data retention and GDPR

Asynchronous processing means the provider stores your inputs and outputs until you fetch them. Anthropic keeps batch request and response data for up to 29 days after creation, lets you delete a finished batch through DELETE /v1/messages/batches/{batch_id}, isolates batches per workspace and allows result downloads in the Console to be disabled; zero data retention eligibility is listed per feature in its data retention page. OpenAI deletes output files 30 days after the batch completes, Gemini keeps results for 6 weeks, and Bedrock writes them to an S3 bucket you control.

For personal data that means three things: delete batches once results are stored, keep the provider and region in your records of processing, and check whether your contract's retention promises cover batch storage at all. The broader picture is in GDPR LLM data residency; if a backfill touches customer records, run it through PII redaction first.

Sources

1. Claude Platform Docs: Batch processing
2. Claude Platform Docs: Rate limits (Message Batches API)
3. Claude Platform Docs: Prompt caching
4. OpenAI API Docs: Batch API
5. OpenAI API Docs: Flex processing
6. OpenAI API Docs: Rate limits (batch queue limits)
7. Google AI for Developers: Gemini Batch API
8. Amazon Bedrock User Guide: Batch inference