Semantic caching for LLM apps: when reusing answers is safe, and when it is not

Support assistants and FAQ bots answer the same questions over and over: how do I reset my password, where is my invoice, what are your delivery times. Every one of those questions costs a model call. A semantic cache stores the answer the first time and returns it when a later question means the same thing, even if it is worded differently. Done well, it removes a large share of calls and answers in milliseconds. Done carelessly, it returns confident answers to questions nobody asked.

This article separates semantic caching from provider prompt caching, shows the architecture, and spends most of its time on the parts that decide whether it is safe: the threshold, cache scope, what not to cache, and invalidation.

How is semantic caching different from prompt caching?

Claude's prompt caching docs are explicit that hits require 100% identical prompt segments and that caching has no effect on output generation. That makes it safe by construction, and you should turn it on first; I cover it in prompt caching and model routing. A semantic cache sits in front of the model and is a different kind of decision, because it changes what the user receives.

How does a semantic cache work?

Libraries such as GPTCache split this into an embedding generator, cache storage for the answers, a vector store for the questions and a similarity evaluator that decides whether a candidate is a match. You do not need a library for it: an embeddings table with a vector index in Postgres, such as pgvector, and twenty lines of code are enough.

# Illustrative semantic cache on Postgres + pgvector
def cached_answer(question: str, scope: dict) -> str:
    q = normalize(question)
    vec = embed(q)                                       # same model for writes and reads
    row = db.fetchone("""
        SELECT answer, 1 - (embedding <=> %(v)s) AS sim
        FROM answer_cache
        WHERE tenant = %(tenant)s AND lang = %(lang)s AND data_version = %(ver)s
          AND expires_at > now()
        ORDER BY embedding <=> %(v)s LIMIT 1""", {"v": vec, **scope})
    if row and row.sim >= THRESHOLD:
        metrics.hit(row.sim)
        return row.answer
    answer = call_model(question, scope)
    if cacheable(question, answer):
        db.execute("INSERT INTO answer_cache (...) VALUES (...)")   # with expires_at = now() + ttl
    return answer

How much does a semantic cache save?

The GPT Semantic Cache paper stored query embeddings in Redis and tested repetitive, customer-service style queries. It reports up to 68.8% fewer API calls, cache hit rates between 61.6% and 68.8% depending on the category, and positive hit rates, hits that returned a correct answer, above 97%. Those numbers come from traffic chosen to be repetitive. A product FAQ looks like that; an open-ended assistant or a coding tool does not, and will see a small fraction of the hits.

Before building anything, measure it: log a week of questions, embed them, and count how many have a near neighbor that a human would accept as the same question. That number is your ceiling.

How do you choose the similarity threshold?

Embedding similarity measures topic, not intent. How do I cancel my order? and How do I cancel my subscription? are very close in vector space and need different answers. Can I return opened items? and Can I return unopened items? differ by one prefix and have opposite answers. A threshold tuned on average similarity will happily serve one for the other.

1. Collect a few hundred pairs of real questions and label each pair: same answer or different answer. Include the tricky negatives deliberately.
2. Compute the similarity of each pair with the embedding model you will use in production.
3. Pick the lowest threshold at which wrong hits stay at or near zero, then accept whatever hit rate that gives you.
4. Re-run the check whenever you change the embedding model, because similarity scales differ between models.

A second check helps at the margin: for hits just above the threshold, ask a small, cheap model whether the two questions have the same answer. That costs far less than generating the answer and removes most of the near misses.

How do you keep a semantic cache from leaking data?

A cache is shared memory between users. If the answer to what is my current discount? is cached without the customer id in the key, the next customer who asks gets the first one's discount. Scope every lookup by everything the answer depends on:

• Tenant and, for personal answers, user. Or exclude personal answers from the cache entirely.
• Language, so a German question does not receive a cached English answer.
• Role or permission level, if internal staff see more than customers.
• Data and prompt version: a new price list, policy text or system prompt should start a fresh cache namespace.

Cached answers containing personal data are personal data at rest, with the retention duties that implies; see PII redaction in LLM pipelines. A cache is also a write path: if users can get arbitrary model output stored and served to others, a prompt-injected answer can spread. Only store answers from trusted flows, and never from sessions flagged by your guardrails.

How do you expire and invalidate cached answers?

Use three mechanisms together. A TTL on every entry, measured in hours or days rather than months. A version in the cache key for the data and prompt the answer came from, so a content update switches to a fresh namespace instead of deleting rows one by one. And a manual purge by document: when someone edits the returns policy, delete every entry whose answer cited that document. That last one requires storing the source ids with each cached answer, which is worth doing anyway for citations, as in grounding and citations.

What should you monitor?

Trace cache hits like any other step, with the similarity score and the id of the cached entry, so a wrong answer can be traced back to the question it was originally generated for; see observability for LLM agents.

Sources

1. Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (2024)
2. Zilliz: GPTCache README (modules, similarity evaluation, eviction)
3. Claude Platform Docs: Prompt caching
4. pgvector README: vector types and indexes