Prompt caching and model routing: cutting LLM cost and latency

Prompt caching is the single biggest cost lever in a production LLM feature, and it is mostly wasted in applications that send the same large prefix on every request. Routing, batching and a per-feature cost dashboard complete the picture. This article explains the mechanism, the numbers to check against as of September 2026, and the order in which to apply them.

The short version: a long, stable prefix in front of your prompt is billed at a fraction of the price; the rest of the cost is decided by which model runs and how many times a day. Get the prefix stable, get the model smaller, measure the result per feature, and most cost problems stop being interesting.

Where does the money actually go?

In almost every application I have costed, input tokens dominate, and within input tokens the system prompt plus the tool definitions plus the conversation history repeat on every call. A support assistant with 12,000 tokens of instructions and tools, asked 2,000 questions a day, sends 24 million identical tokens a day. Nothing about that number is work.

The second cost driver is model choice, and it is a bigger lever than people expect. As of September 2026 the published prices per million input and output tokens are Claude Opus 5.5 at $4 and $20, Claude Fable 5.1 at $10 and $50, Claude Sonnet 5 at $2 and $10, and Claude Haiku 4.5 at $1 and $5 with a 200K context window. A feature that runs on Opus 5.5 and passes its evals on Sonnet 5 has just cut its input cost by half and its output cost by half, with no change to the prompt.

How prompt caching works

Prompt caching is prefix caching. The provider stores an exact byte sequence of your prompt prefix and, when a later request starts with the same sequence, serves that part from storage instead of reprocessing it. The prefix must be identical, which turns out to be the entire engineering task: most "cache misses" in production are requests that differ by a timestamp, a user ID, a random seed or a reordered tool list.

Claude's prompt caching documentation describes up to four cache breakpoints, a lookback over the last 20 blocks, and a minimum of 512 tokens on the 5.x models. Writes are charged at a premium: 1.25 times the input price for the five-minute cache, 2 times for the one-hour cache. Reads come back at 0.1 times the input price, and 0.05 times on Opus 5.5. So a cached prefix on Sonnet 5 costs $0.20 per million tokens instead of $2.

Two invalidation rules cause most of the pain. Editing a single character in the middle of the prefix invalidates everything after it, so prefixes are append-only in practice. And changing the tool definitions invalidates the entire cache, which is why a nightly tool-schema change can quietly double a feature's cost until someone notices.

The ordering rule that follows from this is the one to memorize: tools, then system prompt, then history, then the request. Put a per-request value such as the current timestamp above the history and you push the entire conversation out of the cache on every call.

Choosing a TTL: five minutes or one hour

The five-minute cache is written at 1.25 times the input price and the one-hour cache at 2 times, then read at 0.1 times (0.05 on Opus 5.5). The arithmetic decides the choice. If your prefix is re-read more than about five times within the window, the five-minute cache wins, because the write premium is repaid after the fifth read. For an interactive feature with a burst of questions on the same document, that is the normal case.

The one-hour cache is for a different shape of workload: a prefix that is stable but used rarely, such as a large documentation set behind a feature with a handful of daily users, or a nightly batch job. Paying 2 times once to avoid reprocessing 512 tokens an hour is worthwhile; paying it for a token stream that is re-read every ten seconds is not.

OpenAI's prompt caching guide is the other side of the same coin: caching there is automatic from 1,024 tokens, reads are billed at 0.1 times on GPT-5.6 and later, and cached content is retained for 30 minutes. You get no breakpoints, so ordering is the only lever you control there, and the 30-minute window makes long-lived caches impossible by design.

Routing to the smallest model that works

The second lever is not paying for the biggest model on every request. The pattern is: run the cheapest model that can do the job, and escalate only when it is not confident enough. That requires a confidence signal, and for anything that is not a plain classification it is easier to get one from a separate, small, typed decision model than to parse prose for hedging. That is the approach I use in production and describe in typed decisions for routing and triage: a small calibrated model answers whether the cheap path is good enough, and the expensive model runs when it says no.

The rules that make routing work are unglamorous. Define the escalation condition before you measure anything, so you cannot rationalize a threshold afterwards. Log the rate of escalations: a rising number means the cheap model or the prompt changed, not that the traffic got harder. And keep the answer contract identical across models, otherwise a downgrade becomes a behavior change your users will notice.

Batching work that can wait

The third lever applies to everything that is not user-facing. Batch APIs accept many requests, run them over a longer window and bill at roughly half the price, which is a large discount for summarization, classification, extraction and evaluation runs. The cost is latency: a batch job is measured in hours, not milliseconds.

The practical split is simple. Anything the user is waiting for runs synchronously, with caching and routing applied. Anything that can be queued runs as a batch: nightly summaries, tagging of new documents, scoring an eval suite before a release, enriching a backlog. On the Claude side the discount is 50% of the standard price, so a nightly job that processes 50 million input tokens moves from a large line item to a modest one.

Agent loops are the case where this gets interesting, because a long agent loop resends its whole history on every iteration. Caching the prefix makes each iteration cheap; if the loop can be restructured so independent steps run as one batch instead of sequential turns, the saving is larger still.

A cost dashboard per feature

Per-request cost tells you almost nothing; the number that matters is cost per successful outcome, split by feature. Four metrics per feature are enough to start: input and output tokens per request, cache hit rate, p95 latency, and the share of requests that escalated to the larger model. A feature that doubled its cache hit rate and doubled its escalation rate is not the win it looks like in the token column.

Read the dashboard weekly against the eval suite, not against last week's numbers. A cost reduction that quietly changes output quality is a regression, and the only way to know is to run the same graded set on both configurations. The evals article covers how to build that suite without spending a week on it.

Trade-offs and when not to bother

Prompt caching is not free of complexity. A cache-friendly prompt has to be byte-stable, which constrains personalization and any dynamic content; the invalidation rules are counter-intuitive the first time you hit them; and the savings only materialize once a prefix is long enough to matter, which on Claude means at least 512 tokens. Below that, route instead: spend the effort on choosing a smaller model.

Routing has the opposite trade-off: it adds a second model, a second set of failure modes and an extra hop of latency, and it only pays off when the cheap path is right often enough. Batch APIs are worth it when volume exists, and pointless for a low-traffic feature. If your feature serves 50 requests a day, none of this matters much, and the better use of the same afternoon is the feature itself.

LLM cost and latency checklist

1. Log tokens per request split into cached input, fresh input and output, per feature.
2. Order the prompt tools, system, history, request, and keep everything variable below the history.
3. Mark cache breakpoints explicitly and check the hit rate daily; a silent drop is a schema or template change.
4. Pick the TTL from the read pattern: five minutes for bursty interactive use, one hour for rare but expensive prefixes.
5. Treat tool-definition changes as cost events, because they invalidate the whole cache.
6. Run the cheapest model first and escalate on a confidence signal you defined in advance.
7. Move non-interactive work to batch APIs and accept the hours of latency.
8. Track escalation rate and p95 latency next to cost, so a cheap regression cannot hide.
9. Re-run the eval suite on any configuration change before you celebrate the saving.

All of this belongs in the same conversation as the eval suite and the agent loop design; if you are building a feature that needs all three, the AI engineering page describes how I sequence the work.

Sources

1. Claude API docs: Prompt caching
2. OpenAI API docs: Prompt caching
3. Claude API docs: Models overview, context windows and prices (as of September 2026)