> How semantic caches reuse LLM answers for similar questions, how they differ from prompt caching, and how to avoid wrong hits, leaks and stale answers.
>
> Web page: https://balazscsorba.com/blog/semantic-caching-llm-responses · Language: English · Also available in: [Deutsch](https://balazscsorba.com/de/blog/semantic-caching-llm-responses.md) · [Magyar](https://balazscsorba.com/hu/blog/semantic-caching-llm-responses.md)
> Author: Balázs Csorba · Published: 2026-10-16 · Keywords: semantic caching llm, semantic cache, llm response cache, gptcache, semantic cache similarity threshold, prompt caching vs semantic caching, reduce llm api cost, cache llm answers redis, semantic cache false positive, llm cache invalidation

[Blog](https://balazscsorba.com/blog)/LLMOps & evals

# Semantic caching for LLM apps: when reusing answers is safe, and when it is not

How semantic caches reuse LLM answers for similar questions, how they differ from prompt caching, and how to avoid wrong hits, leaks and stale answers.

[Balázs Csorba](https://balazscsorba.com/about)·October 16, 2026·6 min read

-   Semantic cache
-   LLM cost
-   Latency
-   Embeddings
-   Redis

![A new question is embedded and compared with cached questions; above a similarity threshold the stored answer is returned, below it the model is called.](https://balazscsorba.com/images/blog/semantic-caching-llm-responses/cover.webp?v=c4ac35bd49)

## Key takeaways

-   Prompt caching reuses an identical prefix inside the provider and never changes the answer; a semantic cache returns a stored answer for a similar question.
-   In a published customer-service style test, a semantic cache cut API calls by up to 68.8% with positive hit rates above 97%.
-   The similarity threshold is a product decision: set it from labelled pairs of questions that must and must not share an answer.
-   Scope cache keys by tenant, user, language and data version, or one customer can receive another customer's answer.
-   Cache only answers that do not depend on fresh data or personal context, and give every entry a TTL and an invalidation path.

On this page

1.  [How is semantic caching different from prompt caching?](https://balazscsorba.com/#prompt-vs-semantic-caching)
2.  [How does a semantic cache work?](https://balazscsorba.com/#how-it-works)
3.  [How much does a semantic cache save?](https://balazscsorba.com/#how-much-it-saves)
4.  [How do you choose the similarity threshold?](https://balazscsorba.com/#choosing-the-threshold)
5.  [How do you keep a semantic cache from leaking data?](https://balazscsorba.com/#scope-and-privacy)
6.  [How do you expire and invalidate cached answers?](https://balazscsorba.com/#invalidation)
7.  [What should you monitor?](https://balazscsorba.com/#measure-it)
8.  [Sources](https://balazscsorba.com/#sources)

Listen to this article

0:000:00

Support assistants and FAQ bots answer the same questions over and over: how do I reset my password, where is my invoice, what are your delivery times. Every one of those questions costs a model call. A semantic cache stores the answer the first time and returns it when a later question means the same thing, even if it is worded differently. Done well, it removes a large share of calls and answers in milliseconds. Done carelessly, it returns confident answers to questions nobody asked.

This article separates semantic caching from provider prompt caching, shows the architecture, and spends most of its time on the parts that decide whether it is safe: the threshold, cache scope, what not to cache, and invalidation.

## How is semantic caching different from prompt caching?

Aspect

Prompt caching (provider)

Semantic caching (yours)

Match

Exactly identical prefix

Similar meaning above a threshold

What is reused

Processed input tokens

The whole stored answer

Model still runs

Yes, the answer is freshly generated

No, on a hit

Risk of a wrong answer

None from caching

Yes, on a false hit

Claude's [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) docs are explicit that hits require 100% identical prompt segments and that caching has no effect on output generation. That makes it safe by construction, and you should turn it on first; I cover it in [prompt caching and model routing](https://balazscsorba.com/blog/llm-cost-latency-prompt-caching-routing). A semantic cache sits in front of the model and is a different kind of decision, because it changes what the user receives.

## How does a semantic cache work?

On a miss the model answers and the result is written back, so the cache fills from real traffic.

Libraries such as [GPTCache](https://github.com/zilliztech/GPTCache) split this into an embedding generator, cache storage for the answers, a vector store for the questions and a similarity evaluator that decides whether a candidate is a match. You do not need a library for it: an embeddings table with a vector index in Postgres, such as [pgvector](https://balazscsorba.com/tools/pgvector), and twenty lines of code are enough.

```
# Illustrative semantic cache on Postgres + pgvector
def cached_answer(question: str, scope: dict) -> str:
    q = normalize(question)
    vec = embed(q)                                       # same model for writes and reads
    row = db.fetchone("""
        SELECT answer, 1 - (embedding <=> %(v)s) AS sim
        FROM answer_cache
        WHERE tenant = %(tenant)s AND lang = %(lang)s AND data_version = %(ver)s
          AND expires_at > now()
        ORDER BY embedding <=> %(v)s LIMIT 1""", {"v": vec, **scope})
    if row and row.sim >= THRESHOLD:
        metrics.hit(row.sim)
        return row.answer
    answer = call_model(question, scope)
    if cacheable(question, answer):
        db.execute("INSERT INTO answer_cache (...) VALUES (...)")   # with expires_at = now() + ttl
    return answer
```

## How much does a semantic cache save?

The [GPT Semantic Cache paper](https://arxiv.org/abs/2411.05276) stored query embeddings in Redis and tested repetitive, customer-service style queries. It reports up to 68.8% fewer API calls, cache hit rates between 61.6% and 68.8% depending on the category, and positive hit rates, hits that returned a correct answer, above 97%. Those numbers come from traffic chosen to be repetitive. A product FAQ looks like that; an open-ended assistant or a coding tool does not, and will see a small fraction of the hits.

Before building anything, measure it: log a week of questions, embed them, and count how many have a near neighbor that a human would accept as the same question. That number is your ceiling.

## How do you choose the similarity threshold?

Embedding similarity measures topic, not intent. _How do I cancel my order?_ and _How do I cancel my subscription?_ are very close in vector space and need different answers. _Can I return opened items?_ and _Can I return unopened items?_ differ by one prefix and have opposite answers. A threshold tuned on average similarity will happily serve one for the other.

1.  Collect a few hundred pairs of real questions and label each pair: same answer or different answer. Include the tricky negatives deliberately.
2.  Compute the similarity of each pair with the embedding model you will use in production.
3.  Pick the lowest threshold at which wrong hits stay at or near zero, then accept whatever hit rate that gives you.
4.  Re-run the check whenever you change the embedding model, because similarity scales differ between models.

A second check helps at the margin: for hits just above the threshold, ask a small, cheap model whether the two questions have the same answer. That costs far less than generating the answer and removes most of the near misses.

## How do you keep a semantic cache from leaking data?

A cache is shared memory between users. If the answer to _what is my current discount?_ is cached without the customer id in the key, the next customer who asks gets the first one's discount. Scope every lookup by everything the answer depends on:

-   **Tenant and, for personal answers, user.** Or exclude personal answers from the cache entirely.
-   **Language,** so a German question does not receive a cached English answer.
-   **Role or permission level,** if internal staff see more than customers.
-   **Data and prompt version:** a new price list, policy text or system prompt should start a fresh cache namespace.

Cached answers containing personal data are personal data at rest, with the retention duties that implies; see [PII redaction in LLM pipelines](https://balazscsorba.com/blog/pii-redaction-llm-pipelines). A cache is also a write path: if users can get arbitrary model output stored and served to others, a prompt-injected answer can spread. Only store answers from trusted flows, and never from sessions flagged by your guardrails.

**Never cache answers that depend on live data**

Stock levels, prices, order status and anything behind a tool call change faster than any sensible TTL. Cache the stable parts, such as retrieved policy chunks or tool results with their own expiry, and let the model compose the final answer.

## How do you expire and invalidate cached answers?

Use three mechanisms together. A TTL on every entry, measured in hours or days rather than months. A version in the cache key for the data and prompt the answer came from, so a content update switches to a fresh namespace instead of deleting rows one by one. And a manual purge by document: when someone edits the returns policy, delete every entry whose answer cited that document. That last one requires storing the source ids with each cached answer, which is worth doing anyway for citations, as in [grounding and citations](https://balazscsorba.com/blog/llm-hallucination-grounding-citations).

## What should you monitor?

Metric

Why

Hit rate per scope

Shows where the cache pays off

Similarity distribution of hits

Many hits near the threshold mean risk

Thumbs-down rate on cached vs fresh answers

The direct signal of wrong hits

Sampled human review of hits

Catches wrong hits users did not report

Trace cache hits like any other step, with the similarity score and the id of the cached entry, so a wrong answer can be traced back to the question it was originally generated for; see [observability for LLM agents](https://balazscsorba.com/blog/agent-observability-opentelemetry).

## Sources

1.  [Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (2024)](https://arxiv.org/abs/2411.05276)
2.  [Zilliz: GPTCache README (modules, similarity evaluation, eviction)](https://github.com/zilliztech/GPTCache)
3.  [Claude Platform Docs: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
4.  [pgvector README: vector types and indexes](https://github.com/pgvector/pgvector)

## Frequently asked questions

What is the difference between prompt caching and semantic caching?

Prompt caching is a provider feature that reuses the processed prefix of a prompt when it is exactly identical; the model still generates a new answer. Semantic caching is your own layer that returns a previously generated answer when a new question is similar enough to a cached one, skipping the model call entirely.

How much can a semantic cache save?

It depends on how repetitive the traffic is. The GPT Semantic Cache paper reports up to 68.8% fewer API calls on repetitive customer-service style queries, with cache hit rates of 61.6% to 68.8% and positive hit rates above 97%. Open-ended chat sees far fewer hits.

What similarity threshold should I use?

There is no universal number; it depends on the embedding model and the domain. Collect pairs of real questions labelled as same answer or different answer, and pick the threshold that keeps wrong hits near zero on that set.

Is a semantic cache safe with personal data?

Only if cache entries are scoped to the user or tenant they belong to, and answers that contain personal data are either excluded or cached per user. A shared cache that ignores scope can leak one user's answer to another.

When should I not use a semantic cache?

When answers depend on live data such as stock, prices or order status, on the conversation so far, or on the user's permissions. In those cases cache the expensive parts, such as retrieval results or tool outputs, not the final answer.

Written by Balázs Csorba

Senior fullstack & AI engineer in Styria, Austria – 10+ years of Vue, Nuxt, Node.js and PHP, now building tooling for AI agents.

[AI engineering & MCP servers →](https://balazscsorba.com/expertise/ai-engineer)[About me →](https://balazscsorba.com/about)

## More articles

-   [Structured outputs with JSON Schema: what the guarantee covers, and what it does not](https://balazscsorba.com/blog/structured-outputs-json-schema-llm)
-   [Evaluating AI agents: grading trajectories, outcomes and reliability](https://balazscsorba.com/blog/evaluating-ai-agents-trajectories)
-   [LLM model migrations without regressions: deprecations, evals, shadow traffic](https://balazscsorba.com/blog/llm-model-migration-regression-testing)
-   [Self-hosting LLMs for GDPR: when it is required and what it costs](https://balazscsorba.com/blog/self-hosted-llm-gdpr-cost)

## Sounds like what you need?

Tell me about your project or role – I’d love to hear from you.

[Get in touch](mailto:contact@balazscsorba.com) [Connect on LinkedIn](https://www.linkedin.com/in/balazs-csorba)