LLM rate limits in production: retries, backoff and provider fallback

Every LLM feature that reaches real traffic eventually meets a 429. Sometimes it is your own fault, a batch job that shares a key with the chat endpoint. Sometimes it is the provider, overloaded at peak hours. Either way the user sees a spinner, then an error, and the usual reaction is to wrap the call in a retry loop and move on. That fixes the demo and creates the next outage.

This guide covers what actually happens behind a rate limit, which errors deserve a retry and which never do, how to back off without synchronising every client, why retries belong in exactly one layer, and how to build a fallback chain to another deployment or provider without silently changing what your feature returns. The numbers come from the Anthropic and OpenAI documentation as of October 2026.

How provider rate limits actually work

Anthropic measures Messages API limits in requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM), per model class, and uses a token bucket: capacity refills continuously up to the maximum instead of resetting at the top of each minute. The documentation adds a detail that explains many puzzling 429s: a limit of 60 RPM may be enforced as one request per second, so a short burst fails even if the minute as a whole stays under the limit.

OpenAI counts requests and tokens per minute and per day, plus images per minute, and whichever is hit first triggers the error. Two further details matter for design. OpenAI counts a request against the token limit as the maximum of max_tokens and its estimate of the request size, so a generous max_tokens eats headroom; Anthropic says the opposite for its output limit, which counts only generated tokens. And OpenAI notes that unsuccessful requests still count toward the per-minute limit, which is why tight retry loops make a 429 worse rather than better.

Anthropic also applies acceleration limits: if your organization ramps up usage sharply, you can get 429s even below your tier limits, and the advice is to ramp traffic gradually. That matters on launch day and when a backfill job starts at full speed.

Classify the error before you retry

A retry is only useful when the next attempt can succeed without anything changing. Most retry bugs come from treating every non-200 the same. The Claude API error list makes the split fairly clean:

Streaming adds one more case. With server-sent events an error can arrive after the API has already returned 200, so status-code based retry logic never sees it. Whether to retry a half-streamed answer is a product decision: for a chat UI, show what arrived and offer a retry button; for a background job, discard the partial output and retry the whole request.

Exponential backoff with jitter, and a deadline

The OpenAI guide and the AWS Builders Library agree on the shape: wait a little, then longer after each failure, add randomness so clients do not retry in lockstep, and stop after a limit. OpenAI adds that Retry-After should be treated as a minimum, not a suggestion. Per HTTP semantics the header holds either a number of seconds or an HTTP date, so parse both.

// One retry wrapper for the whole app. Turn SDK retries off (maxRetries: 0) when you use it.
const RETRYABLE = new Set([408, 429, 500, 502, 503, 504, 529])

function retryAfterMs(h: Headers): number | null {
  const v = h.get('retry-after')
  if (!v) return null
  const s = Number(v)
  if (!Number.isNaN(s)) return s * 1000
  const t = Date.parse(v)              // HTTP-date form
  return Number.isNaN(t) ? null : Math.max(0, t - Date.now())
}

export async function withRetry<T>(call: () => Promise<Response>, parse: (r: Response) => Promise<T>,
  { attempts = 4, baseMs = 500, capMs = 20_000, deadlineMs = 45_000 } = {}): Promise<T> {
  const start = Date.now()
  for (let i = 0; ; i++) {
    const res = await call()
    if (res.ok) return parse(res)
    const body = await res.clone().json().catch(() => ({}))
    const spendCap = body?.error?.details?.error_code === 'enforced_spend_limit_reached'
    if (!RETRYABLE.has(res.status) || spendCap || i + 1 >= attempts) throw new Error(`LLM ${res.status}`)
    const backoff = Math.random() * Math.min(capMs, baseMs * 2 ** i)   // "full jitter"
    const wait = Math.max(retryAfterMs(res.headers) ?? 0, backoff)
    if (Date.now() - start + wait > deadlineMs) throw new Error(`LLM ${res.status}: retry budget spent`)
    await new Promise((r) => setTimeout(r, wait))
  }
}

The deadline matters as much as the attempt count. A chat request that is still retrying after 45 seconds is a failure whatever happens next, so give up early and let the fallback chain or the UI handle it. Background jobs can afford longer deadlines, but they should also back off much harder, because they compete with your interactive traffic for the same limits.

Retry in exactly one layer

The AWS article has the clearest warning: with a five-deep call stack where each layer retries three times, the load on the bottom dependency can grow 243 times. LLM stacks hide the same trap. The Anthropic SDK retries connection errors, 429s and 5xx twice by default and honours retry-after. The Vercel AI SDK has maxRetries with a default of 2. A gateway such as LiteLLM has num_retries. If your own code also loops, one user click can turn into dozens of upstream calls during an incident, exactly when the provider is least able to absorb them.

• Pick the layer that knows the most. Usually the gateway, if you run one, because it sees all traffic and every deployment. Otherwise your own wrapper.
• Set retries to zero everywhere else. maxRetries: 0 in the SDK client, no loops in route handlers.
• Add a retry budget. AWS limits local retries with a token bucket: retries flow freely while tokens last, then drop to a fixed rate. That keeps an outage from doubling your request volume.
• Make side effects idempotent first. An agent step that sends an email or writes an order must not run twice because the model call after it was retried.

Building a fallback chain

Retries help with blips. When a model stays unavailable or your limit for it is exhausted, you need somewhere else to send the request. I order the options by how much they change the output: the same model on another deployment or region first, a sibling model from the same provider next, a different provider last, and a graceful degradation (cached answer, simpler feature, honest error) at the end.

You rarely need to write this yourself. LiteLLM takes a fallbacks list in router_settings that maps a model group to ordered backups and fires after num_retries attempts. It also has context_window_fallbacks for prompts that are too long and content_policy_fallbacks for refusals by a provider filter. OpenRouter accepts a models array in priority order, tries the next model when one returns an error, including rate limits, downtime and context-length errors, and bills you for the model that actually handled the request.

# LiteLLM proxy: retries live here, nowhere else
router_settings:
  num_retries: 2
  allowed_fails: 3        # failures per minute before a deployment cools down
  cooldown_time: 30       # seconds the deployment is skipped
  retry_policy:
    RateLimitErrorRetries: 3
    BadRequestErrorRetries: 0
    AuthenticationErrorRetries: 0
    DefaultRetries: 2
  fallbacks: [{"support-primary": ["support-secondary"]}]
  enable_pre_call_checks: true   # needed for context-window fallbacks to check input size

Why fallbacks silently change your product

A fallback that returns a 200 can still be a failure. Prompts tuned for one model often behave differently on another; structured output support and JSON Schema coverage vary between providers; tool-calling formats and refusal styles differ; and prompt caches are per provider, so the first minutes on a fallback cost more and run slower. Teams discover this when the incident is over and support tickets mention odd answers.

• Evaluate the fallback like a migration. Run your regression set against every model in the chain, as described in LLM model migrations without regressions.
• Keep a prompt per model. A fallback entry is a model plus its prompt and parameters, not just a model name.
• Validate output regardless of source. Schema validation and business checks run on every response, which structured outputs do not make optional.
• Record which model answered. Put the serving model in your traces so quality dashboards do not blend two products into one line.
• Check data residency. Falling back from an EU region to a US endpoint can break the promise in your DPA; see GDPR LLM data residency.

Cooldowns and circuit breakers

Without memory, every request rediscovers that a deployment is down and pays the timeout. Martin Fowler's circuit breaker fixes that with three states: closed (calls pass and failures are counted), open (calls fail immediately without touching the dependency) and half-open (after a wait, one trial call decides whether to close again). LiteLLM ships a simpler version per deployment: after allowed_fails failures in a minute (default 3) a deployment is cooled down for cooldown_time seconds (default 5) and traffic goes to the others.

The AWS article is more cautious about breakers: they add modal behaviour that is hard to test and can slow recovery, which is why AWS prefers local retry budgets. My compromise is short cooldowns per deployment, a retry budget per client, and no global breaker that can take a whole feature offline because of one noisy minute.

Raise the ceiling before you hit it

The cheapest 429 is the one that never happens. On most Claude models, cache reads do not count toward ITPM, so caching raises effective throughput: Anthropic's example is a 2,000,000 ITPM limit with an 80% cache hit rate handling 10,000,000 total input tokens per minute. The setup is described in prompt caching and model routing. A few more levers:

• Separate workloads. Anthropic workspaces can have their own lower limits, so a backfill cannot starve the chat endpoint. Separate keys or projects do the same elsewhere.
• Move offline work off the synchronous API. Batch endpoints have their own limits and do not consume the Messages API limits.
• Read the headers. anthropic-ratelimit-*-remaining and x-ratelimit-remaining-* let a client throttle itself before the provider does.
• Ramp up gradually. Start backfills slowly to stay clear of acceleration limits, and protect public endpoints from abuse, see denial of wallet.

Timeouts and long requests

A retry policy without a timeout is incomplete, because a hung connection never produces an error to retry. AWS suggests deriving timeouts from downstream latency percentiles rather than guessing. For LLM calls that means separate budgets for time to first token and total duration. Anthropic recommends streaming or the Batches API for requests that may run past ten minutes, since some networks drop idle connections, and its SDKs refuse non-streaming requests that are expected to exceed a ten-minute timeout. The AI SDK accepts a timeout object with totalMs, stepMs and chunkMs, which maps neatly onto those budgets.

Log the request-id header with every failure. When an incident is the provider's, that id is what their support asks for, and it turns a vague "the API was slow" into something they can trace. Put retries, fallbacks and the serving model into the same trace, as in observability for LLM agents, and you can see afterwards how much of an incident your chain absorbed.

Sources

1. Claude Platform Docs: Rate limits
2. Claude Platform Docs: API errors
3. OpenAI API Docs: Rate limits
4. AWS Builders Library: Timeouts, retries and backoff with jitter
5. MDN: Retry-After header
6. Martin Fowler: Circuit Breaker (2014)
7. LiteLLM Docs: Router, retries and cooldowns
8. LiteLLM Docs: Proxy fallbacks and reliability
9. OpenRouter Docs: Model fallbacks
10. AI SDK Docs: Settings (maxRetries, timeout)