> vLLM turns a Hugging Face checkpoint into an OpenAI-compatible server. What PagedAttention and continuous batching buy, and what running it actually costs.
>
> Web page: https://balazscsorba.com/blog/vllm · Language: English · Also available in: [Deutsch](https://balazscsorba.com/de/blog/vllm.md) · [Magyar](https://balazscsorba.com/hu/blog/vllm.md)
> Author: Balázs Csorba · Published: 2026-06-30 · Keywords: vllm, vllm vs sglang, vllm vs tensorrt-llm, self-hosted llm server, openai compatible api server, llm inference throughput, pagedattention

[Blog](https://balazscsorba.com/blog)/LLMOps & evals

# vLLM reviewed for self-hosted inference

vLLM turns a Hugging Face checkpoint into an OpenAI-compatible server. What PagedAttention and continuous batching buy, and what running it actually costs.

[Balázs Csorba](https://balazscsorba.com/about)·June 30, 2026·11 min read

-   Self-hosted inference
-   OpenAI API
-   PagedAttention
-   GPU serving
-   LLM runtime

![A divided tray of loose sheets feeding one narrow growing column, with an arrow leaving.](https://balazscsorba.com/images/blog/vllm/cover.webp?v=46f6c02794)

## Key takeaways

-   vLLM is the default self-hosted serving engine for open-weight models on NVIDIA and AMD hardware, chosen for breadth of model, quantisation and API coverage rather than for being the fastest.
-   PagedAttention and continuous batching are the reason it exists; the durable claim is that paged allocation cut KV cache waste from 60 to 80 per cent down to under 4 per cent, not the 24x throughput multiple from 2023.
-   Prefix caching is enabled by default and only shortens prefill, so workloads with long repeated prompts gain the most and long generations with unique prompts gain nothing from it.
-   The production failure mode is preemption by recompute when the KV cache is undersized; watch KV cache usage and the cumulative preemption count rather than aggregate throughput.
-   A four-GPU deployment runs six processes and needs at least six physical CPU cores, which is the most common reason throughput lands below expectations.

On this page

1.  [What it actually is](https://balazscsorba.com/#what-it-is)
2.  [How it works](https://balazscsorba.com/#how-it-works)
3.  [Getting a server up](https://balazscsorba.com/#getting-started)
4.  [What the throughput claims are worth](https://balazscsorba.com/#performance)
5.  [Running it in production](https://balazscsorba.com/#running-in-production)
6.  [Where it stings](https://balazscsorba.com/#where-it-stings)
7.  [Verdict](https://balazscsorba.com/#verdict)
8.  [Sources](https://balazscsorba.com/#sources)

Listen to this article

0:000:00

vLLM is an open-source inference engine that turns a Hugging Face checkpoint into an HTTP server that speaks the OpenAI API. It is the layer between a model and everything that calls it, and its entire job is to keep the GPU busy. The verdict up front: on NVIDIA or AMD hardware running open-weight models it is the default choice, because no other project covers this much of the surface with this little glue code. On a laptop, on a CPU-only host, or with one model and one user, it is far too much machinery.

It competes in the serving layer, not the model layer. The alternatives are SGLang, which shares most of its design; NVIDIA's TensorRT-LLM, which trades breadth for peak numbers on NVIDIA parts; llama.cpp, which starts where vLLM gives up; and Hugging Face TGI, whose last release was v3.3.7 in December 2025. It is also the usual substitute for a hosted API, because the same OpenAI client code that talks to a provider can be pointed at a local process.

## What it actually is

The facts that matter when picking a serving engine, all of them taken from the project's own documentation rather than from a vendor page:

-   **Apache-2.0**, with no paid tier and no control plane hosted by the vendor.
-   Started at UC Berkeley's Sky Computing Lab; the documentation credits more than 2,000 contributors and calls it one of the most active open-source AI projects.
-   More than 200 model architectures on Hugging Face, spanning decoder-only, mixture-of-experts, hybrid attention, multimodal, embedding, rerank and reward models.
-   An OpenAI-compatible server plus Anthropic Messages and Cohere embed and rerank endpoints, and it also covers speech and structured output. One model per server process.
-   CUDA and ROCm as first-class targets, Intel XPU and Google TPU supported, with hardware plugins for Ascend NPUs, Gaudi, Spyre, Apple Silicon, MetaX and others.
-   Quantisation across FP8, NVFP4, MXFP4, INT8 and INT4, plus GPTQ, AWQ, GGUF and compressed-tensors checkpoints.
-   A minor release roughly every two weeks: v0.24.0 shipped on 29 June 2026, six weeks after v0.20.2 on 10 May.

## How it works

Two ideas carry most of the throughput. PagedAttention stores the key and value cache in fixed-size blocks and maps them onto non-contiguous GPU memory through a block table, the way an operating system maps pages; the 2023 paper measured 60 to 80 per cent of KV cache wasted through fragmentation and over-reservation in earlier systems, against under 4 per cent for paged allocation. Continuous batching then admits new requests at every decode step instead of waiting for a batch to fill, which is where most of the latency improvement comes from.

![One request through the vLLM V1 engine Clients call an API server process, which tokenises the prompt and forwards it over a socket to an engine core process. The engine core holds the scheduler and the KV cache manager, which hands blocks to one worker process per GPU. Token by token the workers write into paged KV cache blocks and streamed output travels back along the same path.](https://balazscsorba.com/images/blog/vllm/diagram-7.webp?v=13c97d2cdb)

vLLM V1 splits the work across processes: HTTP and tokenisation in the API server, scheduling and cache management in the engine core, one worker per GPU.

The V1 engine, which replaced V0 during 2025, mixes prefill and decode in the same step and gives decode priority, so a long prompt no longer stalls streaming traffic. Chunked prefill is on by default. Prefix caching is on by default too, hashing token blocks so a repeated system prompt is prefilled once; the project's own documentation is careful to note that it only shortens prefill and does nothing for decode, which means it buys almost nothing on long generations with no shared prefix. Speculative decoding is available through n-gram, EAGLE and DFlash proposers rather than a separate small draft model.

## Getting a server up

Installation is one command, and the documented platform is Linux with Python 3.10 to 3.13. Serving one model looks like this:

```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

# Prefix caching and chunked prefill are on by default in V1.
vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 2 \
  --api-key "$VLLM_TOKEN"
```

The flags are mostly capacity decisions. `--max-model-len` caps context, and therefore how much KV cache a single request can hold, so set it to the longest prompt the product actually needs rather than the model maximum. `--gpu-memory-utilization` is the fraction of VRAM pre-allocated to weights and cache; anything left over after the weights is what the start-up profiling pass measures and turns into blocks. `-O0` through `-O3` control how hard the engine compiles and captures CUDA graphs, with `-O2` as the default; `--enforce-eager` skips both entirely. Talking to it needs no new client:

```
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

stream = client.chat.completions.create(
    model="qwen3-8b",
    messages=[
        {"role": "system", "content": "Answer in one sentence."},
        {"role": "user", "content": "Why beat a contiguous KV cache?"},
    ],
    max_tokens=200,
    temperature=0,
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

**Sampling defaults come from the model, not from you**

By default the server applies the `generation_config.json` from the Hugging Face repository, so the model author's recommended sampling values quietly override what the request asks for. Pass `--generation-config vllm` to get the engine's own defaults, and pin sampling in the client when output has to stay stable across upgrades.

## What the throughput claims are worth

The famous numbers come from the 2023 launch post, not from recent benchmarks: LLaMA-7B on an A10G and LLaMA-13B on an A100 40GB, with request lengths sampled from ShareGPT, giving up to 24 times the throughput of Hugging Face Transformers and 2.2 to 3.5 times that of TGI. Those figures are old enough to be history rather than a specification, and the durable part of the story is the memory waste, not the multiples. What has changed since is that the floor moved: the serious competitors all adopted paged caches and in-flight batching, so the useful question is no longer who batches better but who tunes, patches and adds model support faster.

Knob

What it changes

Move it when

`--max-model-len`

Caps context, and with it the KV cache a single request can hold

The longest real prompt is far below the model maximum

`--gpu-memory-utilization`

Fraction of VRAM pre-allocated to weights and KV cache

The start-up log reports a low block count, or requests start preempting

`max_num_batched_tokens`

Prefill tokens per step: small values favour inter-token latency, large values favour time to first token

Interactive chat versus offline batch work

`--tensor-parallel-size`

Splits weights across GPUs, which frees KV cache room on each

The model does not fit, or the KV cache is the binding constraint

`-O0` to `-O3`

Compilation and CUDA graph capture; `-O2` is the default

Boot time matters more than steady-state decode

`--enforce-eager`

Skips compilation and graph capture entirely

Development loops, or measuring how much of a boot is capture

The failure mode that actually turns up in production is preemption. When the KV cache cannot hold every running sequence, vLLM preempts requests and recomputes them from the prompt once space returns, which barely registers in aggregate throughput and dominates tail latency. V1 defaults to recompute rather than swap precisely because swapping cost more, so the cure is capacity rather than a flag.

**Recognise it in the log**

The scheduler warns with `Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space`. Set `disable_log_stats=False` to log the cumulative preemption count, or alert on `vllm:kv_cache_usage_perc` sitting near 1.0. Raising utilisation or tensor parallelism helps; pushing utilisation too far is how a co-tenant on the same host turns into an out-of-memory kill.

## Running it in production

### Observability comes first

The engine exposes a Prometheus endpoint at `/metrics` under a `vllm:` prefix. The metrics design document is unusually explicit about the intent: server-level gauges are there to explain the request-level histograms, and the request-level histograms are the series an operator is meant to alert on.

-   `vllm:time_to_first_token_seconds` — prefill cost, what a user feels on a cold prompt
-   `vllm:inter_token_latency_seconds` — decode speed, what a user feels once generation has started
-   `vllm:e2e_request_latency_seconds` — the series a timeout rule should be written against
-   `vllm:kv_cache_usage_perc` and `vllm:num_requests_running` — capacity; when both sit at their limits together, the queue is growing
-   `vllm:prefix_cache_queries` against `vllm:prefix_cache_hits` — the ratio says whether shared prompts are actually reused, and therefore whether this workload suits the engine

### Process count and CPU

A four-GPU deployment is not one process. V1 runs one API server process, one engine core process and one worker process per GPU — six in total for a single node at tensor parallel size four — and a data-parallel deployment adds a coordinator on top. The tuning guide puts the floor at 2 plus N physical cores for N GPUs, because the engine core runs a busy loop and degrades visibly under CPU starvation.

**Starved CPU looks like a slow GPU**

The guide names tokenisation, scheduling latency and streaming detokenisation as what suffers first, with lower-than-expected GPU utilisation as the symptom. With hyperthreading enabled, budget twice (2 + N) in vCPUs.

### Security posture

Authentication is one shared secret. The `--api-key` flag, or the `VLLM_API_KEY` environment variable, turns on a header check and accepts several keys at once so they can be rotated. There is no user model, no per-tenant quota and no authorisation layer, so anything that can reach the port can use the whole GPU.

**Development endpoints are a production incident waiting**

Setting `VLLM_SERVER_DEV_MODE=1` registers endpoints including `/pause`, `/reset_prefix_cache` and `/collective_rpc`, and the documentation carries its own security warning about them. Keep the port on a trusted network and put authentication, quotas and rate limiting in front of it.

## Where it stings

Three things first. It is a GPU server, not a universal runtime: the documented target is Linux with CUDA or ROCm, so a CPU-only host or an Apple Silicon machine means a different project and a different model format. Boot time is real, because the default optimisation level compiles the model and captures CUDA graphs, and a cold container can spend minutes in the compiler before the first token. And the API is OpenAI-shaped rather than OpenAI-complete: the `suffix` parameter is unsupported, the `user` parameter is ignored, and parallel tool calls are best-effort and model-dependent.

Engine

Licence

Where it wins

What it costs you

**vLLM**

Apache-2.0

Breadth: the most architectures, the widest quantisation and hardware target set, one API surface for text, embeddings, rerank, speech and structured output

Compilation and graph capture on every boot, one model per server, and a busy loop that needs CPU behind it

SGLang

Apache-2.0

Radix-style prefix sharing and multi-turn state reuse, which suits agent and retrieval traffic that hits the same long context repeatedly

A smaller model zoo and a thinner serving surface outside chat completions

TensorRT-LLM

Apache-2.0, NVIDIA stack only

Peak numbers on the newest NVIDIA parts, FP4 and FP8 kernels, first-class integration with Dynamo and Triton

NVIDIA only, and a rebuild whenever the model, the quantisation or the GPU generation changes

llama.cpp

MIT

CPU, Apple Silicon and edge hardware, GGUF quantisation, a single binary with no Python runtime

A different model format, a weaker batched-serving story, no comparable surface for embeddings or rerank

For most teams the real comparison is the first row against the second. vLLM and SGLang solve the same problem with the same primitives, both Apache-2.0, both OpenAI-compatible, and both will serve an ordinary chat workload well. SGLang's prefix tree is the better fit when the same long context is hit over and over; vLLM's model coverage and quantisation matrix are the better fit when new checkpoints arrive faster than workloads repeat.

## Verdict

vLLM is the tool to reach for by default, and the reason is not raw speed. It is surface area: the most models, the most quantisation formats, the most hardware targets, and one API that covers completions, chat, embeddings, rerank, speech and structured output. The costs are real and mostly boring to fix. Boot time is tunable, preemption is a capacity problem, and a starved CPU is a deployment mistake. None of them is a reason to pick something else.

1.  Pick it when you serve open-weight models on NVIDIA or AMD GPUs and the workload is batched: many concurrent requests rather than one at a time.
2.  Pick it when model turnover is high, because a new checkpoint usually runs before the alternatives support it.
3.  Pick it when the surrounding stack already speaks the OpenAI API, since the migration is a base URL.
4.  Skip it for a laptop, an edge box or a single-user tool; llama.cpp or an MLX runtime does the job in a fraction of the memory.
5.  Skip it when one model on one NVIDIA generation is committed and peak tokens per second is the only goal, because TensorRT-LLM will win that trade at the cost of the lock-in.
6.  Benchmark SGLang before committing if the traffic is agentic or retrieval-heavy with heavy context reuse; the two are close enough that the deciding factor is usually which one the team can debug at 3am.

**The opinionated part**

The case for vLLM has quietly become a case about breadth rather than speed, and the projects that overtake it are unlikely to win on batching. They will win by being better at one workload. Teams are usually better served by one general engine plus one specialist than by an engine per workload, and that is the argument for standardising on vLLM even when a rival is measurably faster for a single traffic shape.

## Sources

1.  [vLLM documentation](https://docs.vllm.ai/en/latest/)
2.  [Optimization and tuning for the V1 engine](https://docs.vllm.ai/en/stable/configuration/optimization.html)
3.  [Architecture overview and the V1 process layout](https://docs.vllm.ai/en/stable/design/arch_overview.html)
4.  [Metrics design](https://docs.vllm.ai/en/stable/design/metrics.html)
5.  [Automatic prefix caching and its documented limits](https://docs.vllm.ai/en/stable/features/automatic_prefix_caching.html)
6.  [Online serving and the HTTP API surface](https://docs.vllm.ai/en/stable/serving/online_serving.html)
7.  [Inside vLLM: anatomy of a high-throughput inference system](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm)
8.  [vLLM: easy, fast and cheap LLM serving with PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html)
9.  [Release history](https://github.com/vllm-project/vllm/releases)

## Frequently asked questions

Is vLLM faster than SGLang or TensorRT-LLM?

Not categorically, and most published comparisons are not comparable. The 2023 vLLM launch post reported up to 24 times the throughput of Hugging Face Transformers and 2.2 to 3.5 times that of TGI, measured on LLaMA-7B and LLaMA-13B with request lengths sampled from ShareGPT. That predates competitors adopting paged caches, so benchmark your own request-length distribution rather than trusting any single multiple.

Do I need a GPU to run vLLM?

Linux with CUDA or ROCm is the documented target, with Python 3.10 to 3.13. Intel XPU and Google TPU are supported separately, and Apple Silicon is covered by a different, MLX-based project rather than by vLLM itself. CPU-only inference is llama.cpp territory, not vLLM's.

How much VRAM does vLLM need?

vLLM takes a configurable fraction of VRAM for weights and KV cache, then measures what is left by running a profiling forward pass at start-up and reports how many KV cache blocks fit. In practice the binding constraint is the model weights at your chosen quantisation; the remainder is KV cache, and how much that is decides your concurrency.

Can one vLLM server host several models?

Not natively. A server process hosts one model at a time, so multi-model deployments run one process per model behind a router, or use data parallelism to replicate the same model across several GPUs. LoRA adapters are the exception: they can be loaded and unloaded at runtime, and the documentation restricts that to local development.

How do I keep vLLM output reproducible across upgrades?

Pin the engine version, since minor releases land roughly every two weeks, and pass --generation-config vllm to stop the server applying the model's generation\_config.json from Hugging Face, which otherwise overrides your sampling defaults silently. For deterministic runs, set temperature to zero and pin the model revision as well.

Written by Balázs Csorba

Senior fullstack & AI engineer in Styria, Austria – 10+ years of Vue, Nuxt, Node.js and PHP, now building tooling for AI agents.

[AI engineering & MCP servers →](https://balazscsorba.com/expertise/ai-engineer)[About me →](https://balazscsorba.com/about)

## More articles

-   [DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent](https://balazscsorba.com/blog/deepseek-v4-1-flash-1017)
-   [LLM rate limits in production: retries, backoff and provider fallback](https://balazscsorba.com/blog/llm-rate-limits-retries-provider-fallback)
-   [Prompt versioning in production: git, registries, labels and rollbacks](https://balazscsorba.com/blog/prompt-versioning-management)
-   [LLM batch APIs: half-price inference for work that can wait](https://balazscsorba.com/blog/llm-batch-api-offline-workloads)

## Sounds like what you need?

Tell me about your project or role – I’d love to hear from you.

[Get in touch](mailto:contact@balazscsorba.com) [Connect on LinkedIn](https://www.linkedin.com/in/balazs-csorba)