[{"data":1,"prerenderedAt":614},["ShallowReactive",2],{"blog-semantic-caching-llm-responses-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":25,"sources":35,"cover":48,"og":49,"expertise":50,"locales":51,"lang":52,"title":55,"description":56,"coverAlt":57,"metaTitle":58,"takeaways":59,"faq":65,"toc":81,"blocks":106,"others":345},"semantic-caching-llm-responses","2026-10-16",6,"llmops",[9,10,11,12,13],"Semantic cache","LLM cost","Latency","Embeddings","Redis",[15,16,17,18,19,20,21,22,23,24],"semantic caching llm","semantic cache","llm response cache","gptcache","semantic cache similarity threshold","prompt caching vs semantic caching","reduce llm api cost","cache llm answers redis","semantic cache false positive","llm cache invalidation",[26,29,32],{"name":27,"url":28},"Cache (computing)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCache_(computing)",{"name":30,"url":31},"Sentence embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSentence_embedding",{"name":33,"url":34},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[36,39,42,45],{"title":37,"url":38},"Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2411.05276",{"title":40,"url":41},"Zilliz: GPTCache README (modules, similarity evaluation, eviction)","https:\u002F\u002Fgithub.com\u002Fzilliztech\u002FGPTCache",{"title":43,"url":44},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching",{"title":46,"url":47},"pgvector README: vector types and indexes","https:\u002F\u002Fgithub.com\u002Fpgvector\u002Fpgvector","\u002Fimages\u002Fblog\u002Fsemantic-caching-llm-responses\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fsemantic-caching-llm-responses\u002Fog.jpg","ai-engineer",[52,53,54],"en","de","hu","Semantic caching for LLM apps: when reusing answers is safe, and when it is not","How semantic caches reuse LLM answers for similar questions, how they differ from prompt caching, and how to avoid wrong hits, leaks and stale answers.","A new question is embedded and compared with cached questions; above a similarity threshold the stored answer is returned, below it the model is called.","Semantic caching for LLM apps: safe reuse · Balázs Csorba",[60,61,62,63,64],"Prompt caching reuses an identical prefix inside the provider and never changes the answer; a semantic cache returns a stored answer for a similar question.","In a published customer-service style test, a semantic cache cut API calls by up to 68.8% with positive hit rates above 97%.","The similarity threshold is a product decision: set it from labelled pairs of questions that must and must not share an answer.","Scope cache keys by tenant, user, language and data version, or one customer can receive another customer's answer.","Cache only answers that do not depend on fresh data or personal context, and give every entry a TTL and an invalidation path.",[66,69,72,75,78],{"q":67,"a":68},"What is the difference between prompt caching and semantic caching?","Prompt caching is a provider feature that reuses the processed prefix of a prompt when it is exactly identical; the model still generates a new answer. Semantic caching is your own layer that returns a previously generated answer when a new question is similar enough to a cached one, skipping the model call entirely.",{"q":70,"a":71},"How much can a semantic cache save?","It depends on how repetitive the traffic is. The GPT Semantic Cache paper reports up to 68.8% fewer API calls on repetitive customer-service style queries, with cache hit rates of 61.6% to 68.8% and positive hit rates above 97%. Open-ended chat sees far fewer hits.",{"q":73,"a":74},"What similarity threshold should I use?","There is no universal number; it depends on the embedding model and the domain. Collect pairs of real questions labelled as same answer or different answer, and pick the threshold that keeps wrong hits near zero on that set.",{"q":76,"a":77},"Is a semantic cache safe with personal data?","Only if cache entries are scoped to the user or tenant they belong to, and answers that contain personal data are either excluded or cached per user. A shared cache that ignores scope can leak one user's answer to another.",{"q":79,"a":80},"When should I not use a semantic cache?","When answers depend on live data such as stock, prices or order status, on the conversation so far, or on the user's permissions. In those cases cache the expensive parts, such as retrieval results or tool outputs, not the final answer.",[82,85,88,91,94,97,100,103],{"id":83,"title":84},"prompt-vs-semantic-caching","How is semantic caching different from prompt caching?",{"id":86,"title":87},"how-it-works","How does a semantic cache work?",{"id":89,"title":90},"how-much-it-saves","How much does a semantic cache save?",{"id":92,"title":93},"choosing-the-threshold","How do you choose the similarity threshold?",{"id":95,"title":96},"scope-and-privacy","How do you keep a semantic cache from leaking data?",{"id":98,"title":99},"invalidation","How do you expire and invalidate cached answers?",{"id":101,"title":102},"measure-it","What should you monitor?",{"id":104,"title":105},"sources","Sources",[107,111,114,117,155,169,170,179,191,194,195,202,205,206,225,237,240,241,248,271,279,286,287,295,296,323,330,331],{"type":108,"content":109},"paragraph",[110],"Support assistants and FAQ bots answer the same questions over and over: how do I reset my password, where is my invoice, what are your delivery times. Every one of those questions costs a model call. A semantic cache stores the answer the first time and returns it when a later question means the same thing, even if it is worded differently. Done well, it removes a large share of calls and answers in milliseconds. Done carelessly, it returns confident answers to questions nobody asked.",{"type":108,"content":112},[113],"This article separates semantic caching from provider prompt caching, shows the architecture, and spends most of its time on the parts that decide whether it is safe: the threshold, cache scope, what not to cache, and invalidation.",{"type":115,"level":116,"id":83,"text":84},"heading",2,{"type":118,"head":119,"rows":126},"table",[120,122,124],[121],"Aspect",[123],"Prompt caching (provider)",[125],"Semantic caching (yours)",[127,134,141,148],[128,130,132],[129],"Match",[131],"Exactly identical prefix",[133],"Similar meaning above a threshold",[135,137,139],[136],"What is reused",[138],"Processed input tokens",[140],"The whole stored answer",[142,144,146],[143],"Model still runs",[145],"Yes, the answer is freshly generated",[147],"No, on a hit",[149,151,153],[150],"Risk of a wrong answer",[152],"None from caching",[154],"Yes, on a false hit",{"type":108,"content":156},[157,158,162,163,168],"Claude's ",{"tag":159,"href":44,"children":160},"a",[161],"prompt caching"," docs are explicit that hits require 100% identical prompt segments and that caching has no effect on output generation. That makes it safe by construction, and you should turn it on first; I cover it in ",{"tag":164,"to":165,"children":166},"link","\u002Fblog\u002Fllm-cost-latency-prompt-caching-routing",[167],"prompt caching and model routing",". A semantic cache sits in front of the model and is a different kind of decision, because it changes what the user receives.",{"type":115,"level":116,"id":86,"text":87},{"type":171,"attrs":172,"inner":176,"caption":177},"diagram",{"viewBox":173,"role":174,"aria-labelledby":175},"0 0 720 170","img","sc1-t sc1-d","\u003Ctitle id=\"sc1-t\">Semantic cache lookup\u003C\u002Ftitle>\u003Cdesc id=\"sc1-d\">The question is normalized and embedded, a vector search finds the nearest cached question within the same scope, and the stored answer is returned if the similarity passes the threshold; otherwise the model is called and the new answer is stored with a TTL.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"sc1-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"20\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"79\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Question\u003C\u002Ftext>\u003Ctext x=\"79\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">normalize\u003C\u002Ftext>\u003Cpath d=\"M138 85H158\" class=\"d-line\" marker-end=\"url(#sc1-ah)\" \u002F>\u003Crect x=\"160\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-accent\" \u002F>\u003Ctext x=\"220\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Embed\u003C\u002Ftext>\u003Ctext x=\"220\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">same model\u003C\u002Ftext>\u003Cpath d=\"M279 85H299\" class=\"d-line\" marker-end=\"url(#sc1-ah)\" \u002F>\u003Crect x=\"301\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"360\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Search\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">within scope\u003C\u002Ftext>\u003Cpath d=\"M419 85H439\" class=\"d-line\" marker-end=\"url(#sc1-ah)\" \u002F>\u003Crect x=\"441\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"500\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Hit?\u003C\u002Ftext>\u003Ctext x=\"500\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">≥ threshold\u003C\u002Ftext>\u003Cpath d=\"M560 85H580\" class=\"d-line\" marker-end=\"url(#sc1-ah)\" \u002F>\u003Crect x=\"582\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"641\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Answer\u003C\u002Ftext>\u003Ctext x=\"641\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">cached or new\u003C\u002Ftext>",[178],"On a miss the model answers and the result is written back, so the cache fills from real traffic.",{"type":108,"content":180},[181,182,185,186,190],"Libraries such as ",{"tag":159,"href":41,"children":183},[184],"GPTCache"," split this into an embedding generator, cache storage for the answers, a vector store for the questions and a similarity evaluator that decides whether a candidate is a match. You do not need a library for it: an embeddings table with a vector index in Postgres, such as ",{"tag":164,"to":187,"children":188},"\u002Ftools\u002Fpgvector",[189],"pgvector",", and twenty lines of code are enough.",{"type":192,"code":193},"code","# Illustrative semantic cache on Postgres + pgvector\ndef cached_answer(question: str, scope: dict) -> str:\n    q = normalize(question)\n    vec = embed(q)                                       # same model for writes and reads\n    row = db.fetchone(\"\"\"\n        SELECT answer, 1 - (embedding \u003C=> %(v)s) AS sim\n        FROM answer_cache\n        WHERE tenant = %(tenant)s AND lang = %(lang)s AND data_version = %(ver)s\n          AND expires_at > now()\n        ORDER BY embedding \u003C=> %(v)s LIMIT 1\"\"\", {\"v\": vec, **scope})\n    if row and row.sim >= THRESHOLD:\n        metrics.hit(row.sim)\n        return row.answer\n    answer = call_model(question, scope)\n    if cacheable(question, answer):\n        db.execute(\"INSERT INTO answer_cache (...) VALUES (...)\")   # with expires_at = now() + ttl\n    return answer\n",{"type":115,"level":116,"id":89,"text":90},{"type":108,"content":196},[197,198,201],"The ",{"tag":159,"href":38,"children":199},[200],"GPT Semantic Cache paper"," stored query embeddings in Redis and tested repetitive, customer-service style queries. It reports up to 68.8% fewer API calls, cache hit rates between 61.6% and 68.8% depending on the category, and positive hit rates, hits that returned a correct answer, above 97%. Those numbers come from traffic chosen to be repetitive. A product FAQ looks like that; an open-ended assistant or a coding tool does not, and will see a small fraction of the hits.",{"type":108,"content":203},[204],"Before building anything, measure it: log a week of questions, embed them, and count how many have a near neighbor that a human would accept as the same question. That number is your ceiling.",{"type":115,"level":116,"id":92,"text":93},{"type":108,"content":207},[208,209,213,214,217,218,213,221,224],"Embedding similarity measures topic, not intent. ",{"tag":210,"children":211},"em",[212],"How do I cancel my order?"," and ",{"tag":210,"children":215},[216],"How do I cancel my subscription?"," are very close in vector space and need different answers. ",{"tag":210,"children":219},[220],"Can I return opened items?",{"tag":210,"children":222},[223],"Can I return unopened items?"," differ by one prefix and have opposite answers. A threshold tuned on average similarity will happily serve one for the other.",{"type":226,"ordered":227,"items":228},"list",true,[229,231,233,235],[230],"Collect a few hundred pairs of real questions and label each pair: same answer or different answer. Include the tricky negatives deliberately.",[232],"Compute the similarity of each pair with the embedding model you will use in production.",[234],"Pick the lowest threshold at which wrong hits stay at or near zero, then accept whatever hit rate that gives you.",[236],"Re-run the check whenever you change the embedding model, because similarity scales differ between models.",{"type":108,"content":238},[239],"A second check helps at the margin: for hits just above the threshold, ask a small, cheap model whether the two questions have the same answer. That costs far less than generating the answer and removes most of the near misses.",{"type":115,"level":116,"id":95,"text":96},{"type":108,"content":242},[243,244,247],"A cache is shared memory between users. If the answer to ",{"tag":210,"children":245},[246],"what is my current discount?"," is cached without the customer id in the key, the next customer who asks gets the first one's discount. Scope every lookup by everything the answer depends on:",{"type":226,"items":249},[250,256,261,266],[251,255],{"tag":252,"children":253},"strong",[254],"Tenant and, for personal answers, user."," Or exclude personal answers from the cache entirely.",[257,260],{"tag":252,"children":258},[259],"Language,"," so a German question does not receive a cached English answer.",[262,265],{"tag":252,"children":263},[264],"Role or permission level,"," if internal staff see more than customers.",[267,270],{"tag":252,"children":268},[269],"Data and prompt version:"," a new price list, policy text or system prompt should start a fresh cache namespace.",{"type":108,"content":272},[273,274,278],"Cached answers containing personal data are personal data at rest, with the retention duties that implies; see ",{"tag":164,"to":275,"children":276},"\u002Fblog\u002Fpii-redaction-llm-pipelines",[277],"PII redaction in LLM pipelines",". A cache is also a write path: if users can get arbitrary model output stored and served to others, a prompt-injected answer can spread. Only store answers from trusted flows, and never from sessions flagged by your guardrails.",{"type":280,"variant":281,"title":282,"body":283},"callout","warn","Never cache answers that depend on live data",[284],[285],"Stock levels, prices, order status and anything behind a tool call change faster than any sensible TTL. Cache the stable parts, such as retrieved policy chunks or tool results with their own expiry, and let the model compose the final answer.",{"type":115,"level":116,"id":98,"text":99},{"type":108,"content":288},[289,290,294],"Use three mechanisms together. A TTL on every entry, measured in hours or days rather than months. A version in the cache key for the data and prompt the answer came from, so a content update switches to a fresh namespace instead of deleting rows one by one. And a manual purge by document: when someone edits the returns policy, delete every entry whose answer cited that document. That last one requires storing the source ids with each cached answer, which is worth doing anyway for citations, as in ",{"tag":164,"to":291,"children":292},"\u002Fblog\u002Fllm-hallucination-grounding-citations",[293],"grounding and citations",".",{"type":115,"level":116,"id":101,"text":102},{"type":118,"head":297,"rows":302},[298,300],[299],"Metric",[301],"Why",[303,308,313,318],[304,306],[305],"Hit rate per scope",[307],"Shows where the cache pays off",[309,311],[310],"Similarity distribution of hits",[312],"Many hits near the threshold mean risk",[314,316],[315],"Thumbs-down rate on cached vs fresh answers",[317],"The direct signal of wrong hits",[319,321],[320],"Sampled human review of hits",[322],"Catches wrong hits users did not report",{"type":108,"content":324},[325,326,294],"Trace cache hits like any other step, with the similarity score and the id of the cached entry, so a wrong answer can be traced back to the question it was originally generated for; see ",{"tag":164,"to":327,"children":328},"\u002Fblog\u002Fagent-observability-opentelemetry",[329],"observability for LLM agents",{"type":115,"level":116,"id":104,"text":105},{"type":226,"ordered":227,"items":332},[333,336,339,342],[334],{"tag":159,"href":38,"children":335},[37],[337],{"tag":159,"href":41,"children":338},[40],[340],{"tag":159,"href":44,"children":341},[43],[343],{"tag":159,"href":47,"children":344},[46],[346,399,456,510],{"slug":347,"published":5,"minutes":348,"category":7,"tags":349,"keywords":355,"about":367,"sources":374,"cover":393,"og":394,"expertise":50,"locales":395,"lang":52,"title":396,"description":397,"coverAlt":398},"structured-outputs-json-schema-llm",9,[350,351,352,353,354],"Structured outputs","JSON Schema","Constrained decoding","LLM APIs","Validation",[356,357,358,359,360,361,362,363,364,365,366],"structured outputs LLM","JSON schema LLM output","OpenAI structured outputs strict mode","Claude structured outputs","Gemini structured output","constrained decoding","JSON mode vs structured outputs","validate LLM JSON output","Zod structured output","Pydantic LLM output","vLLM structured outputs",[368,371,373],{"name":369,"url":370},"JSON","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FJSON",{"name":351,"url":372},"https:\u002F\u002Fjson-schema.org\u002F",{"name":33,"url":34},[375,378,381,384,387,390],{"title":376,"url":377},"OpenAI API docs: Structured model outputs","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fstructured-outputs",{"title":379,"url":380},"Claude API docs: Structured outputs","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fstructured-outputs",{"title":382,"url":383},"Gemini API docs: Structured outputs","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fstructured-output",{"title":385,"url":386},"vLLM docs: Structured Outputs","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fstructured_outputs.html",{"title":388,"url":389},"Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.02442",{"title":391,"url":392},"Geng et al.: JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.10868","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fstructured-outputs-json-schema-llm\u002Fog.jpg",[52,53,54],"Structured outputs with JSON Schema: what the guarantee covers, and what it does not","How structured outputs and constrained decoding work on OpenAI, Claude, Gemini and vLLM, which schema rules matter, and what you still validate.","A JSON Schema compiled into a grammar that masks tokens, producing schema-valid JSON that then passes a business validation step.",{"slug":400,"published":5,"minutes":348,"category":7,"tags":401,"keywords":407,"about":420,"sources":428,"cover":450,"og":451,"expertise":50,"locales":452,"lang":52,"title":453,"description":454,"coverAlt":455},"evaluating-ai-agents-trajectories",[402,403,404,405,406],"Agent evals","Trajectories","Tool calling","pass^k","LLM-as-judge",[408,409,410,411,412,413,414,415,416,417,418,419],"evaluating AI agents","agent trajectory evaluation","agent evals","tool call accuracy","trajectory match evaluator","pass^k metric","tau-bench","LLM as a judge for agents","agent regression testing","outcome vs trajectory evaluation","agentevals","tool correctness metric",[421,424,427],{"name":422,"url":423},"Intelligent agent","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIntelligent_agent",{"name":425,"url":426},"Software testing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSoftware_testing",{"name":33,"url":34},[429,432,435,438,441,444,447],{"title":430,"url":431},"Anthropic: Demystifying evals for AI agents (Jan 2026)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents",{"title":433,"url":434},"Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045",{"title":436,"url":437},"Barres et al.: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07982",{"title":439,"url":440},"LangChain: agentevals (GitHub)","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals",{"title":442,"url":443},"DeepEval docs: Tool Correctness","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fmetrics-tool-correctness",{"title":445,"url":446},"Ragas docs: Agentic or tool use metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002Fagents\u002F",{"title":448,"url":449},"OpenAI API docs: Trace grading","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ftrace-grading","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fevaluating-ai-agents-trajectories\u002Fog.jpg",[52,53,54],"Evaluating AI agents: grading trajectories, outcomes and reliability","How to evaluate AI agents: outcome vs trajectory graders, trajectory match modes, pass^k, isolated environments and LLM judges that hold up.","An agent trajectory of tool calls graded twice: a trajectory grader checks order and arguments, an outcome grader compares the database with the goal state.",{"slug":457,"published":5,"minutes":458,"category":7,"tags":459,"keywords":465,"about":477,"sources":484,"cover":504,"og":505,"expertise":50,"locales":506,"lang":52,"title":507,"description":508,"coverAlt":509},"llm-model-migration-regression-testing",10,[460,461,462,463,464],"Model migration","Regression testing","Deprecations","LLM evals","Canary releases",[466,467,468,469,470,471,472,473,474,475,476],"LLM model migration","LLM regression testing","model deprecation OpenAI","Claude model deprecation","switch LLM model without breaking","LLM model upgrade checklist","prompt regression testing","shadow testing LLM","model retirement date","golden dataset LLM","LLM drift monitoring",[478,480,481],{"name":461,"url":479},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRegression_testing",{"name":33,"url":34},{"name":482,"url":483},"Feature toggle","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FFeature_toggle",[485,488,491,494,497,500,503],{"title":486,"url":487},"Claude API docs: Model deprecations","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fabout-claude\u002Fmodel-deprecations",{"title":489,"url":490},"OpenAI API docs: Deprecations","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fdeprecations",{"title":492,"url":493},"Gemini API docs: Models and version name patterns","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fmodels",{"title":495,"url":496},"Claude API docs: Claude Opus 5.5 migration guide","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fmodels\u002Fopus-5-5\u002Fmigration-guide",{"title":498,"url":499},"Chen, Zaharia, Zou: How is ChatGPT's behavior changing over time? (2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.09009",{"title":501,"url":502},"Sclar et al.: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.11324",{"title":430,"url":431},"\u002Fimages\u002Fblog\u002Fllm-model-migration-regression-testing\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-model-migration-regression-testing\u002Fog.jpg",[52,53,54],"LLM model migrations without regressions: deprecations, evals, shadow traffic","A playbook for switching LLM models: deprecation timelines, contract fixes, a regression suite, prompt re-tuning, shadow traffic and canary rollout.","Five migration stages from inventory and contract fixes through offline evals and shadow traffic to a canary rollout with a rollback path.",{"slug":511,"published":512,"minutes":458,"category":7,"tags":513,"keywords":519,"about":528,"sources":538,"cover":608,"og":609,"expertise":50,"locales":610,"lang":52,"title":611,"description":612,"coverAlt":613},"self-hosted-llm-gdpr-cost","2026-10-09",[514,515,516,517,518],"Self-hosting","GDPR","vLLM","GPU cost","Open-weight models",[520,521,522,523,524,525,526,527],"self-hosted LLM GDPR","self-host LLM data protection","LLM GPU memory requirements","vLLM throughput benchmark","EU GPU cloud for LLMs","LLM API vs self-hosting break-even","open-weight LLM hardware cost","data processing agreement LLM",[529,532,533,536],{"name":530,"url":531},"General Data Protection Regulation","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGeneral_Data_Protection_Regulation",{"name":33,"url":34},{"name":534,"url":535},"Graphics processing unit","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGraphics_processing_unit",{"name":516,"url":537},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm",[539,542,545,548,551,554,557,560,563,566,569,572,575,578,581,584,587,590,593,596,599,602,605],{"title":540,"url":541},"openai\u002Fgpt-oss-120b model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fopenai\u002Fgpt-oss-120b",{"title":543,"url":544},"mistralai\u002FMistral-Small-3.2-24B-Instruct-2506 model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-Small-3.2-24B-Instruct-2506",{"title":546,"url":547},"meta-llama\u002FLlama-3.3-70B-Instruct model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FLlama-3.3-70B-Instruct",{"title":549,"url":550},"mistralai\u002FMistral-Large-3-675B-Instruct-2512 model card (Hugging Face)","https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-Large-3-675B-Instruct-2512",{"title":552,"url":553},"vLLM blog: v0.6.0 performance update (September 2024)","https:\u002F\u002Fblog.vllm.ai\u002F2024\u002F09\u002F05\u002Fperf-update.html",{"title":555,"url":556},"vLLM documentation: engine arguments","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fconfiguration\u002Fengine_args.html",{"title":558,"url":559},"vLLM documentation: vllm bench serve","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fbench\u002Fserve.html",{"title":561,"url":562},"SemiAnalysis InferenceX: Llama 3.3 70B on H100 vs H200","https:\u002F\u002Finferencex.semianalysis.com\u002Fcompare\u002Fllama-3-3-70b-h100-vs-h200",{"title":564,"url":565},"Scaleway GPU instances pricing","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fpricing\u002Fgpu\u002F",{"title":567,"url":568},"Scaleway blog: a transparent update on Scaleway pricing (27 April 2026)","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fblog\u002Fa-transparent-update-on-scaleway-pricing\u002F",{"title":570,"url":571},"Scaleway Generative APIs pricing","https:\u002F\u002Fwww.scaleway.com\u002Fen\u002Fpricing\u002Fmodel-as-a-service\u002F",{"title":573,"url":574},"OVHcloud public cloud price list","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fprices\u002F",{"title":576,"url":577},"OVHcloud AI Endpoints","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fai-endpoints\u002F",{"title":579,"url":580},"OVHcloud AI Endpoints model catalogue","https:\u002F\u002Fwww.ovhcloud.com\u002Fen\u002Fpublic-cloud\u002Fai-endpoints\u002Fcatalog\u002F",{"title":582,"url":583},"Hetzner dedicated GPU servers","https:\u002F\u002Fwww.hetzner.com\u002Fdedicated-rootserver\u002Fmatrix-gpu\u002F",{"title":585,"url":586},"NVIDIA newsroom: GeForce RTX 50 Series launch","https:\u002F\u002Fnvidianews.nvidia.com\u002Fnews\u002Fnvidia-blackwell-geforce-rtx-50-series-opens-new-world-of-ai-computer-graphics",{"title":588,"url":589},"NVIDIA GeForce RTX 5090","https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fgeforce\u002Fgraphics-cards\u002F50-series\u002Frtx-5090\u002F",{"title":591,"url":592},"NVIDIA GeForce software licence","https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fdrivers\u002Fgeforce-license\u002F",{"title":594,"url":595},"GDPR Article 28: processor","https:\u002F\u002Fgdpr-info.eu\u002Fart-28-gdpr\u002F",{"title":597,"url":598},"GDPR Article 44: general principle for transfers","https:\u002F\u002Fgdpr-info.eu\u002Fart-44-gdpr\u002F",{"title":600,"url":601},"European Commission: EU-US data transfers (Data Privacy Framework)","https:\u002F\u002Fcommission.europa.eu\u002Flaw\u002Flaw-topic\u002Fdata-protection\u002Finternational-dimension-data-protection\u002Feu-us-data-transfers_en",{"title":603,"url":604},"EDPB Opinion 28\u002F2024 on AI models (adopted 17 December 2024)","https:\u002F\u002Fwww.edpb.europa.eu\u002Fsystem\u002Ffiles\u002F2024-12\u002Fedpb_opinion_202428_ai-models_en.pdf",{"title":606,"url":607},"Mistral AI pricing","https:\u002F\u002Fmistral.ai\u002Fpricing","\u002Fimages\u002Fblog\u002Fself-hosted-llm-gdpr-cost\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fself-hosted-llm-gdpr-cost\u002Fog.jpg",[52,53,54],"Self-hosting LLMs for GDPR: when it is required and what it costs","Self-hosting an LLM for GDPR: when it is required, GPU memory for open-weight models, EU prices as of October 2026 and break-even per million tokens.","Cover art for self-hosted LLMs under GDPR: a decision between your own GPUs and an EU API, with a break-even line.",1792177152212]