[{"data":1,"prerenderedAt":756},["ShallowReactive",2],{"blog-vllm-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":21,"sources":25,"cover":53,"og":54,"expertise":55,"locales":56,"lang":57,"title":60,"description":61,"coverAlt":62,"metaTitle":63,"takeaways":64,"faq":70,"toc":86,"blocks":111,"others":513},"vllm","2026-06-30",11,"llmops",[9,10,11,12,13],"Self-hosted inference","OpenAI API","PagedAttention","GPU serving","LLM runtime",[4,15,16,17,18,19,20],"vllm vs sglang","vllm vs tensorrt-llm","self-hosted llm server","openai compatible api server","llm inference throughput","pagedattention",[22],{"name":23,"url":24},"vLLM","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FvLLM",[26,29,32,35,38,41,44,47,50],{"title":27,"url":28},"vLLM documentation","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002F",{"title":30,"url":31},"Optimization and tuning for the V1 engine","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fconfiguration\u002Foptimization.html",{"title":33,"url":34},"Architecture overview and the V1 process layout","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fdesign\u002Farch_overview.html",{"title":36,"url":37},"Metrics design","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fdesign\u002Fmetrics.html",{"title":39,"url":40},"Automatic prefix caching and its documented limits","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Ffeatures\u002Fautomatic_prefix_caching.html",{"title":42,"url":43},"Online serving and the HTTP API surface","https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fserving\u002Fonline_serving.html",{"title":45,"url":46},"Inside vLLM: anatomy of a high-throughput inference system","https:\u002F\u002Fvllm.ai\u002Fblog\u002F2025-09-05-anatomy-of-vllm",{"title":48,"url":49},"vLLM: easy, fast and cheap LLM serving with PagedAttention","https:\u002F\u002Fblog.vllm.ai\u002F2023\u002F06\u002F20\u002Fvllm.html",{"title":51,"url":52},"Release history","https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Freleases","\u002Fimages\u002Fblog\u002Fvllm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fvllm\u002Fog.jpg","ai-engineer",[57,58,59],"en","de","hu","vLLM reviewed for self-hosted inference","vLLM turns a Hugging Face checkpoint into an OpenAI-compatible server. What PagedAttention and continuous batching buy, and what running it actually costs.","A divided tray of loose sheets feeding one narrow growing column, with an arrow leaving.","vLLM reviewed for self-hosted inference · Balázs Csorba",[65,66,67,68,69],"vLLM is the default self-hosted serving engine for open-weight models on NVIDIA and AMD hardware, chosen for breadth of model, quantisation and API coverage rather than for being the fastest.","PagedAttention and continuous batching are the reason it exists; the durable claim is that paged allocation cut KV cache waste from 60 to 80 per cent down to under 4 per cent, not the 24x throughput multiple from 2023.","Prefix caching is enabled by default and only shortens prefill, so workloads with long repeated prompts gain the most and long generations with unique prompts gain nothing from it.","The production failure mode is preemption by recompute when the KV cache is undersized; watch KV cache usage and the cumulative preemption count rather than aggregate throughput.","A four-GPU deployment runs six processes and needs at least six physical CPU cores, which is the most common reason throughput lands below expectations.",[71,74,77,80,83],{"q":72,"a":73},"Is vLLM faster than SGLang or TensorRT-LLM?","Not categorically, and most published comparisons are not comparable. The 2023 vLLM launch post reported up to 24 times the throughput of Hugging Face Transformers and 2.2 to 3.5 times that of TGI, measured on LLaMA-7B and LLaMA-13B with request lengths sampled from ShareGPT. That predates competitors adopting paged caches, so benchmark your own request-length distribution rather than trusting any single multiple.",{"q":75,"a":76},"Do I need a GPU to run vLLM?","Linux with CUDA or ROCm is the documented target, with Python 3.10 to 3.13. Intel XPU and Google TPU are supported separately, and Apple Silicon is covered by a different, MLX-based project rather than by vLLM itself. CPU-only inference is llama.cpp territory, not vLLM's.",{"q":78,"a":79},"How much VRAM does vLLM need?","vLLM takes a configurable fraction of VRAM for weights and KV cache, then measures what is left by running a profiling forward pass at start-up and reports how many KV cache blocks fit. In practice the binding constraint is the model weights at your chosen quantisation; the remainder is KV cache, and how much that is decides your concurrency.",{"q":81,"a":82},"Can one vLLM server host several models?","Not natively. A server process hosts one model at a time, so multi-model deployments run one process per model behind a router, or use data parallelism to replicate the same model across several GPUs. LoRA adapters are the exception: they can be loaded and unloaded at runtime, and the documentation restricts that to local development.",{"q":84,"a":85},"How do I keep vLLM output reproducible across upgrades?","Pin the engine version, since minor releases land roughly every two weeks, and pass --generation-config vllm to stop the server applying the model's generation_config.json from Hugging Face, which otherwise overrides your sampling defaults silently. For deterministic runs, set temperature to zero and pin the model revision as well.",[87,90,93,96,99,102,105,108],{"id":88,"title":89},"what-it-is","What it actually is",{"id":91,"title":92},"how-it-works","How it works",{"id":94,"title":95},"getting-started","Getting a server up",{"id":97,"title":98},"performance","What the throughput claims are worth",{"id":100,"title":101},"running-in-production","Running it in production",{"id":103,"title":104},"where-it-stings","Where it stings",{"id":106,"title":107},"verdict","Verdict",{"id":109,"title":110},"sources","Sources",[112,116,119,122,125,147,148,151,157,160,161,164,167,194,196,211,212,215,281,284,302,303,307,318,352,355,358,364,367,378,398,399,410,456,459,460,463,477,482,483],{"type":113,"content":114},"paragraph",[115],"vLLM is an open-source inference engine that turns a Hugging Face checkpoint into an HTTP server that speaks the OpenAI API. It is the layer between a model and everything that calls it, and its entire job is to keep the GPU busy. The verdict up front: on NVIDIA or AMD hardware running open-weight models it is the default choice, because no other project covers this much of the surface with this little glue code. On a laptop, on a CPU-only host, or with one model and one user, it is far too much machinery.",{"type":113,"content":117},[118],"It competes in the serving layer, not the model layer. The alternatives are SGLang, which shares most of its design; NVIDIA's TensorRT-LLM, which trades breadth for peak numbers on NVIDIA parts; llama.cpp, which starts where vLLM gives up; and Hugging Face TGI, whose last release was v3.3.7 in December 2025. It is also the usual substitute for a hosted API, because the same OpenAI client code that talks to a provider can be pointed at a local process.",{"type":120,"level":121,"id":88,"text":89},"heading",2,{"type":113,"content":123},[124],"The facts that matter when picking a serving engine, all of them taken from the project's own documentation rather than from a vendor page:",{"type":126,"ordered":127,"items":128},"list",false,[129,135,137,139,141,143,145],[130,134],{"tag":131,"children":132},"strong",[133],"Apache-2.0",", with no paid tier and no control plane hosted by the vendor.",[136],"Started at UC Berkeley's Sky Computing Lab; the documentation credits more than 2,000 contributors and calls it one of the most active open-source AI projects.",[138],"More than 200 model architectures on Hugging Face, spanning decoder-only, mixture-of-experts, hybrid attention, multimodal, embedding, rerank and reward models.",[140],"An OpenAI-compatible server plus Anthropic Messages and Cohere embed and rerank endpoints, and it also covers speech and structured output. One model per server process.",[142],"CUDA and ROCm as first-class targets, Intel XPU and Google TPU supported, with hardware plugins for Ascend NPUs, Gaudi, Spyre, Apple Silicon, MetaX and others.",[144],"Quantisation across FP8, NVFP4, MXFP4, INT8 and INT4, plus GPTQ, AWQ, GGUF and compressed-tensors checkpoints.",[146],"A minor release roughly every two weeks: v0.24.0 shipped on 29 June 2026, six weeks after v0.20.2 on 10 May.",{"type":120,"level":121,"id":91,"text":92},{"type":113,"content":149},[150],"Two ideas carry most of the throughput. PagedAttention stores the key and value cache in fixed-size blocks and maps them onto non-contiguous GPU memory through a block table, the way an operating system maps pages; the 2023 paper measured 60 to 80 per cent of KV cache wasted through fragmentation and over-reservation in earlier systems, against under 4 per cent for paged allocation. Continuous batching then admits new requests at every decode step instead of waiting for a batch to fill, which is where most of the latency improvement comes from.",{"type":152,"src":153,"alt":154,"wide":155,"caption":156},"image","\u002Fimages\u002Fblog\u002Fvllm\u002Fdiagram-7.webp","One request through the vLLM V1 engine Clients call an API server process, which tokenises the prompt and forwards it over a socket to an engine core process. The engine core holds the scheduler and the KV cache manager, which hands blocks to one worker process per GPU. Token by token the workers write into paged KV cache blocks and streamed output travels back along the same path.",true,"vLLM V1 splits the work across processes: HTTP and tokenisation in the API server, scheduling and cache management in the engine core, one worker per GPU.",{"type":113,"content":158},[159],"The V1 engine, which replaced V0 during 2025, mixes prefill and decode in the same step and gives decode priority, so a long prompt no longer stalls streaming traffic. Chunked prefill is on by default. Prefix caching is on by default too, hashing token blocks so a repeated system prompt is prefilled once; the project's own documentation is careful to note that it only shortens prefill and does nothing for decode, which means it buys almost nothing on long generations with no shared prefix. Speculative decoding is available through n-gram, EAGLE and DFlash proposers rather than a separate small draft model.",{"type":120,"level":121,"id":94,"text":95},{"type":113,"content":162},[163],"Installation is one command, and the documented platform is Linux with Python 3.10 to 3.13. Serving one model looks like this:",{"type":165,"code":166},"code","uv venv --python 3.12 --seed\nsource .venv\u002Fbin\u002Factivate\nuv pip install vllm --torch-backend=auto\n\n# Prefix caching and chunked prefill are on by default in V1.\nvllm serve Qwen\u002FQwen3-8B \\\n  --served-model-name qwen3-8b \\\n  --max-model-len 32768 \\\n  --gpu-memory-utilization 0.9 \\\n  --tensor-parallel-size 2 \\\n  --api-key \"$VLLM_TOKEN\"",{"type":113,"content":168},[169,170,173,174,177,178,181,182,185,186,189,190,193],"The flags are mostly capacity decisions. ",{"tag":165,"children":171},[172],"--max-model-len"," caps context, and therefore how much KV cache a single request can hold, so set it to the longest prompt the product actually needs rather than the model maximum. ",{"tag":165,"children":175},[176],"--gpu-memory-utilization"," is the fraction of VRAM pre-allocated to weights and cache; anything left over after the weights is what the start-up profiling pass measures and turns into blocks. ",{"tag":165,"children":179},[180],"-O0"," through ",{"tag":165,"children":183},[184],"-O3"," control how hard the engine compiles and captures CUDA graphs, with ",{"tag":165,"children":187},[188],"-O2"," as the default; ",{"tag":165,"children":191},[192],"--enforce-eager"," skips both entirely. Talking to it needs no new client:",{"type":165,"code":195},"from openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"EMPTY\",\n    base_url=\"http:\u002F\u002Flocalhost:8000\u002Fv1\",\n)\n\nstream = client.chat.completions.create(\n    model=\"qwen3-8b\",\n    messages=[\n        {\"role\": \"system\", \"content\": \"Answer in one sentence.\"},\n        {\"role\": \"user\", \"content\": \"Why beat a contiguous KV cache?\"},\n    ],\n    max_tokens=200,\n    temperature=0,\n    stream=True,\n)\n\nfor chunk in stream:\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)",{"type":197,"variant":198,"title":199,"body":200},"callout","note","Sampling defaults come from the model, not from you",[201],[202,203,206,207,210],"By default the server applies the ",{"tag":165,"children":204},[205],"generation_config.json"," from the Hugging Face repository, so the model author's recommended sampling values quietly override what the request asks for. Pass ",{"tag":165,"children":208},[209],"--generation-config vllm"," to get the engine's own defaults, and pin sampling in the client when output has to stay stable across upgrades.",{"type":120,"level":121,"id":97,"text":98},{"type":113,"content":213},[214],"The famous numbers come from the 2023 launch post, not from recent benchmarks: LLaMA-7B on an A10G and LLaMA-13B on an A100 40GB, with request lengths sampled from ShareGPT, giving up to 24 times the throughput of Hugging Face Transformers and 2.2 to 3.5 times that of TGI. Those figures are old enough to be history rather than a specification, and the durable part of the story is the memory waste, not the multiples. What has changed since is that the floor moved: the serious competitors all adopted paged caches and in-flight batching, so the useful question is no longer who batches better but who tunes, patches and adds model support faster.",{"type":216,"head":217,"rows":224},"table",[218,220,222],[219],"Knob",[221],"What it changes",[223],"Move it when",[225,233,241,250,259,273],[226,229,231],[227],{"tag":165,"children":228},[172],[230],"Caps context, and with it the KV cache a single request can hold",[232],"The longest real prompt is far below the model maximum",[234,237,239],[235],{"tag":165,"children":236},[176],[238],"Fraction of VRAM pre-allocated to weights and KV cache",[240],"The start-up log reports a low block count, or requests start preempting",[242,246,248],[243],{"tag":165,"children":244},[245],"max_num_batched_tokens",[247],"Prefill tokens per step: small values favour inter-token latency, large values favour time to first token",[249],"Interactive chat versus offline batch work",[251,255,257],[252],{"tag":165,"children":253},[254],"--tensor-parallel-size",[256],"Splits weights across GPUs, which frees KV cache room on each",[258],"The model does not fit, or the KV cache is the binding constraint",[260,266,271],[261,263,264],{"tag":165,"children":262},[180]," to ",{"tag":165,"children":265},[184],[267,268,270],"Compilation and CUDA graph capture; ",{"tag":165,"children":269},[188]," is the default",[272],"Boot time matters more than steady-state decode",[274,277,279],[275],{"tag":165,"children":276},[192],[278],"Skips compilation and graph capture entirely",[280],"Development loops, or measuring how much of a boot is capture",{"type":113,"content":282},[283],"The failure mode that actually turns up in production is preemption. When the KV cache cannot hold every running sequence, vLLM preempts requests and recomputes them from the prompt once space returns, which barely registers in aggregate throughput and dominates tail latency. V1 defaults to recompute rather than swap precisely because swapping cost more, so the cure is capacity rather than a flag.",{"type":197,"variant":285,"title":286,"body":287},"warn","Recognise it in the log",[288],[289,290,293,294,297,298,301],"The scheduler warns with ",{"tag":165,"children":291},[292],"Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space",". Set ",{"tag":165,"children":295},[296],"disable_log_stats=False"," to log the cumulative preemption count, or alert on ",{"tag":165,"children":299},[300],"vllm:kv_cache_usage_perc"," sitting near 1.0. Raising utilisation or tensor parallelism helps; pushing utilisation too far is how a co-tenant on the same host turns into an out-of-memory kill.",{"type":120,"level":121,"id":100,"text":101},{"type":120,"level":304,"id":305,"text":306},3,"observability","Observability comes first",{"type":113,"content":308},[309,310,313,314,317],"The engine exposes a Prometheus endpoint at ",{"tag":165,"children":311},[312],"\u002Fmetrics"," under a ",{"tag":165,"children":315},[316],"vllm:"," prefix. The metrics design document is unusually explicit about the intent: server-level gauges are there to explain the request-level histograms, and the request-level histograms are the series an operator is meant to alert on.",{"type":126,"ordered":127,"items":319},[320,325,330,335,343],[321,324],{"tag":165,"children":322},[323],"vllm:time_to_first_token_seconds"," — prefill cost, what a user feels on a cold prompt",[326,329],{"tag":165,"children":327},[328],"vllm:inter_token_latency_seconds"," — decode speed, what a user feels once generation has started",[331,334],{"tag":165,"children":332},[333],"vllm:e2e_request_latency_seconds"," — the series a timeout rule should be written against",[336,338,339,342],{"tag":165,"children":337},[300]," and ",{"tag":165,"children":340},[341],"vllm:num_requests_running"," — capacity; when both sit at their limits together, the queue is growing",[344,347,348,351],{"tag":165,"children":345},[346],"vllm:prefix_cache_queries"," against ",{"tag":165,"children":349},[350],"vllm:prefix_cache_hits"," — the ratio says whether shared prompts are actually reused, and therefore whether this workload suits the engine",{"type":120,"level":304,"id":353,"text":354},"process-and-cpu","Process count and CPU",{"type":113,"content":356},[357],"A four-GPU deployment is not one process. V1 runs one API server process, one engine core process and one worker process per GPU — six in total for a single node at tensor parallel size four — and a data-parallel deployment adds a coordinator on top. The tuning guide puts the floor at 2 plus N physical cores for N GPUs, because the engine core runs a busy loop and degrades visibly under CPU starvation.",{"type":197,"variant":359,"title":360,"body":361},"tip","Starved CPU looks like a slow GPU",[362],[363],"The guide names tokenisation, scheduling latency and streaming detokenisation as what suffers first, with lower-than-expected GPU utilisation as the symptom. With hyperthreading enabled, budget twice (2 + N) in vCPUs.",{"type":120,"level":304,"id":365,"text":366},"security","Security posture",{"type":113,"content":368},[369,370,373,374,377],"Authentication is one shared secret. The ",{"tag":165,"children":371},[372],"--api-key"," flag, or the ",{"tag":165,"children":375},[376],"VLLM_API_KEY"," environment variable, turns on a header check and accepts several keys at once so they can be rotated. There is no user model, no per-tenant quota and no authorisation layer, so anything that can reach the port can use the whole GPU.",{"type":197,"variant":285,"title":379,"body":380},"Development endpoints are a production incident waiting",[381],[382,383,386,387,390,391,338,394,397],"Setting ",{"tag":165,"children":384},[385],"VLLM_SERVER_DEV_MODE=1"," registers endpoints including ",{"tag":165,"children":388},[389],"\u002Fpause",", ",{"tag":165,"children":392},[393],"\u002Freset_prefix_cache",{"tag":165,"children":395},[396],"\u002Fcollective_rpc",", and the documentation carries its own security warning about them. Keep the port on a trusted network and put authentication, quotas and rate limiting in front of it.",{"type":120,"level":121,"id":103,"text":104},{"type":113,"content":400},[401,402,405,406,409],"Three things first. It is a GPU server, not a universal runtime: the documented target is Linux with CUDA or ROCm, so a CPU-only host or an Apple Silicon machine means a different project and a different model format. Boot time is real, because the default optimisation level compiles the model and captures CUDA graphs, and a cold container can spend minutes in the compiler before the first token. And the API is OpenAI-shaped rather than OpenAI-complete: the ",{"tag":165,"children":403},[404],"suffix"," parameter is unsupported, the ",{"tag":165,"children":407},[408],"user"," parameter is ignored, and parallel tool calls are best-effort and model-dependent.",{"type":216,"head":411,"rows":420},[412,414,416,418],[413],"Engine",[415],"Licence",[417],"Where it wins",[419],"What it costs you",[421,430,438,447],[422,425,426,428],[423],{"tag":131,"children":424},[23],[133],[427],"Breadth: the most architectures, the widest quantisation and hardware target set, one API surface for text, embeddings, rerank, speech and structured output",[429],"Compilation and graph capture on every boot, one model per server, and a busy loop that needs CPU behind it",[431,433,434,436],[432],"SGLang",[133],[435],"Radix-style prefix sharing and multi-turn state reuse, which suits agent and retrieval traffic that hits the same long context repeatedly",[437],"A smaller model zoo and a thinner serving surface outside chat completions",[439,441,443,445],[440],"TensorRT-LLM",[442],"Apache-2.0, NVIDIA stack only",[444],"Peak numbers on the newest NVIDIA parts, FP4 and FP8 kernels, first-class integration with Dynamo and Triton",[446],"NVIDIA only, and a rebuild whenever the model, the quantisation or the GPU generation changes",[448,450,452,454],[449],"llama.cpp",[451],"MIT",[453],"CPU, Apple Silicon and edge hardware, GGUF quantisation, a single binary with no Python runtime",[455],"A different model format, a weaker batched-serving story, no comparable surface for embeddings or rerank",{"type":113,"content":457},[458],"For most teams the real comparison is the first row against the second. vLLM and SGLang solve the same problem with the same primitives, both Apache-2.0, both OpenAI-compatible, and both will serve an ordinary chat workload well. SGLang's prefix tree is the better fit when the same long context is hit over and over; vLLM's model coverage and quantisation matrix are the better fit when new checkpoints arrive faster than workloads repeat.",{"type":120,"level":121,"id":106,"text":107},{"type":113,"content":461},[462],"vLLM is the tool to reach for by default, and the reason is not raw speed. It is surface area: the most models, the most quantisation formats, the most hardware targets, and one API that covers completions, chat, embeddings, rerank, speech and structured output. The costs are real and mostly boring to fix. Boot time is tunable, preemption is a capacity problem, and a starved CPU is a deployment mistake. None of them is a reason to pick something else.",{"type":126,"ordered":155,"items":464},[465,467,469,471,473,475],[466],"Pick it when you serve open-weight models on NVIDIA or AMD GPUs and the workload is batched: many concurrent requests rather than one at a time.",[468],"Pick it when model turnover is high, because a new checkpoint usually runs before the alternatives support it.",[470],"Pick it when the surrounding stack already speaks the OpenAI API, since the migration is a base URL.",[472],"Skip it for a laptop, an edge box or a single-user tool; llama.cpp or an MLX runtime does the job in a fraction of the memory.",[474],"Skip it when one model on one NVIDIA generation is committed and peak tokens per second is the only goal, because TensorRT-LLM will win that trade at the cost of the lock-in.",[476],"Benchmark SGLang before committing if the traffic is agentic or retrieval-heavy with heavy context reuse; the two are close enough that the deciding factor is usually which one the team can debug at 3am.",{"type":197,"variant":198,"title":478,"body":479},"The opinionated part",[480],[481],"The case for vLLM has quietly become a case about breadth rather than speed, and the projects that overtake it are unlikely to win on batching. They will win by being better at one workload. Teams are usually better served by one general engine plus one specialist than by an engine per workload, and that is the argument for standardising on vLLM even when a rival is measurably faster for a single traffic shape.",{"type":120,"level":121,"id":109,"text":110},{"type":126,"ordered":155,"items":484},[485,489,492,495,498,501,504,507,510],[486],{"tag":487,"href":28,"children":488},"a",[27],[490],{"tag":487,"href":31,"children":491},[30],[493],{"tag":487,"href":34,"children":494},[33],[496],{"tag":487,"href":37,"children":497},[36],[499],{"tag":487,"href":40,"children":500},[39],[502],{"tag":487,"href":43,"children":503},[42],[505],{"tag":487,"href":46,"children":506},[45],[508],{"tag":487,"href":49,"children":509},[48],[511],{"tag":487,"href":52,"children":512},[51],[514,571,639,700],{"slug":515,"published":516,"minutes":517,"category":7,"tags":518,"keywords":524,"about":538,"sources":546,"cover":565,"og":566,"expertise":55,"locales":567,"lang":57,"title":568,"description":569,"coverAlt":570},"deepseek-v4-1-flash-1017","2026-10-17",15,[519,520,521,522,523],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[525,526,527,528,529,530,531,532,533,534,535,536,537],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[539,541,543],{"name":519,"url":540},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":522,"url":542},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":544,"url":545},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[547,550,553,556,559,562],{"title":548,"url":549},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":551,"url":552},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":554,"url":555},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":557,"url":558},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":560,"url":561},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":563,"url":564},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[57,58,59],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":572,"published":516,"minutes":573,"category":7,"tags":574,"keywords":580,"about":592,"sources":602,"cover":633,"og":634,"expertise":55,"locales":635,"lang":57,"title":636,"description":637,"coverAlt":638},"llm-rate-limits-retries-provider-fallback",10,[575,576,577,578,579],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[581,582,583,584,585,586,587,588,589,590,591],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[593,596,599],{"name":594,"url":595},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":597,"url":598},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":600,"url":601},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[603,606,609,612,615,618,621,624,627,630],{"title":604,"url":605},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":607,"url":608},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":610,"url":611},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":613,"url":614},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":616,"url":617},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":619,"url":620},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":622,"url":623},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":625,"url":626},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":628,"url":629},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":631,"url":632},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[57,58,59],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":640,"published":516,"minutes":641,"category":7,"tags":642,"keywords":648,"about":659,"sources":669,"cover":694,"og":695,"expertise":55,"locales":696,"lang":57,"title":697,"description":698,"coverAlt":699},"prompt-versioning-management",8,[643,644,645,646,647],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[649,650,651,652,653,654,655,656,657,658],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[660,663,666],{"name":661,"url":662},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":664,"url":665},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":667,"url":668},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[670,673,676,679,682,685,688,691],{"title":671,"url":672},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":674,"url":675},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":677,"url":678},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":680,"url":681},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":683,"url":684},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":686,"url":687},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":689,"url":690},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":692,"url":693},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[57,58,59],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":701,"published":516,"minutes":702,"category":7,"tags":703,"keywords":708,"about":719,"sources":729,"cover":750,"og":751,"expertise":55,"locales":752,"lang":57,"title":753,"description":754,"coverAlt":755},"llm-batch-api-offline-workloads",9,[704,705,645,706,707],"Batch API","Cost","Prompt caching","Data pipelines",[709,710,711,712,713,714,715,716,717,718],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[720,723,726],{"name":721,"url":722},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":724,"url":725},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":727,"url":728},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[730,733,735,736,739,742,744,747],{"title":731,"url":732},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":734,"url":605},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":692,"url":693},{"title":737,"url":738},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":740,"url":741},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":743,"url":611},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":745,"url":746},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":748,"url":749},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[57,58,59],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156387]