vLLM Production Guide 2026 — PagedAttention, Quantization, and Multi-GPU Deployment

The most commonly cited vLLM benchmark: "23× throughput improvement over naive serving." It's real. It's also not what you need to understand to deploy vLLM in production without shooting yourself in the foot.

That 23× number comes from continuous batching versus static batching under high concurrency and high request length variance. Under low concurrency — which describes most internal tools and early-stage deployments — the gains are real but 3–5×, not 23×. The decisions that actually determine your production performance are quantization format, tensor parallelism config, whether speculative decoding fits your workload, and KV cache headroom. Getting these wrong means either leaving 4× throughput on the table or deploying something that OOMs during a traffic spike.

This guide covers the mechanics behind each decision with the numbers to make them confidently.

PagedAttention — Why Memory Management Is the Core Innovation

vLLM's throughput advantage starts with PagedAttention. Understanding it explains both the performance gains and the most common production OOM failures.

Every LLM generates a KV (Key-Value) cache during inference — stored attention states for all previous tokens. Naive implementations pre-allocate this cache contiguously at request start: you reserve GPU memory for the maximum sequence length whether the request ends at 100 tokens or 4,000.

For Llama 3.1 70B with GQA (Grouped Query Attention):

# KV cache memory per token, per layer
num_kv_heads   = 8       # GQA reduces from 64 attention heads to 8 KV heads
head_dim       = 128
bytes_fp16     = 2
num_layers     = 80

bytes_per_token_per_layer = 2 * num_kv_heads * head_dim * bytes_fp16
# = 2 * 8 * 128 * 2 = 4,096 bytes = 4 KB

bytes_per_token_all_layers = bytes_per_token_per_layer * num_layers
# = 4 KB * 80 = 320 KB per token

# Naive pre-allocation at max_seq_len=4096:
# pre_allocated_per_request = 320 KB * 4096 = ~1.28 GB

With static allocation, 10 concurrent requests pre-allocate 12.8 GB of KV cache — even if most requests end at 200 tokens. That's 10× overallocation.

PagedAttention manages the KV cache like OS virtual memory: a shared pool of fixed-size blocks (typically 16 or 32 tokens each), allocated dynamically as tokens generate and freed immediately when a request completes.

{/* DIAGRAM: KV Cache Memory Layout
    ┌─────────────────────────────────────────────────────────────┐
    │  STATIC ALLOCATION (naive)                                  │
    │  [Req1: 1.28 GB reserved ███░░░░░░░░░░░░░░░░░░░░░░░░░░░]  │
    │  [Req2: 1.28 GB reserved ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  │
    │  [Req3: 1.28 GB reserved █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  │
    │  ░ = unused reserved memory (~60-70% waste)                 │
    ├─────────────────────────────────────────────────────────────┤
    │  PAGEDATTENTION (vLLM)                                      │
    │  Block pool [B1][B2][B3][B4][B5][B6][B7][B8][B9][B10]...  │
    │  Req1 → [B1][B4][B7]        (grows token by token)         │
    │  Req2 → [B2][B5]            (frees B2,B5 on completion)    │
    │  Req3 → [B3][B6][B8][B9]                                   │
    │  ~95% utilization, near-zero fragmentation                  │
    └─────────────────────────────────────────────────────────────┘
*/}

The practical result: 55–80% reduction in memory waste, directly enabling more concurrent sequences on the same hardware. This is what makes continuous batching effective — PagedAttention gives you the memory headroom to keep the batch full.

Continuous Batching — What the 23× Number Actually Means

Static batching processes a full batch start to finish before accepting new requests. If you batch 8 requests and one runs for 2,000 tokens while the other 7 complete at 200 tokens each, those 7 slots sit idle for 1,800 decode steps.

Continuous batching inserts new requests the moment a slot frees — at the iteration level, not the request level:

Static batching (high length variance):
  [Req1 ████████████████████ 2000 tokens]
  [Req2 ██░░░░░░░░░░░░░░░░░░ 200 + 1800 idle steps]
  [Req3 ██░░░░░░░░░░░░░░░░░░ 200 + 1800 idle steps]
  GPU utilization: ~25% on this batch

Continuous batching:
  [Req1 ████████████████████ 2000 tokens]
  [Req2 ██][Req9 ████][Req13 ██][Req17...]
  [Req3 ██][Req10 ██ ][Req14 ██][Req18...]
  GPU utilization: ~92%

The 23× improvement is the extreme scenario: maximum length variance, high concurrency. In a uniform-length workload (all requests ~same token count), continuous batching delivers 2–4× over static. In a mixed production workload with real user queries, 4–8× is a realistic expectation.

vLLM vs Ollama: In direct benchmarks, vLLM delivers approximately 9× higher throughput than Ollama at batch sizes above 8. Ollama is built for local single-user inference — it doesn't implement continuous batching. Don't benchmark them at batch=1 and conclude they're comparable.

Quantization — The Decision Tree

Quantization reduces weight precision to shrink memory footprint and increase throughput. The right choice depends on your GPU generation and quality tolerance.

What GPU?
│
├─ H100 (SXM5 or PCIe 80GB)
│   └─ → FP8 weights + FP8 activations  [DEFAULT for H100]
│        Quality loss: <1% on MMLU, HumanEval, GSM8K
│        Throughput gain: 1.5–2× vs FP16
│        H100 Transformer Engine supports FP8 natively
│        vllm serve <model> --quantization fp8
│
├─ A100 (80GB)
│   ├─ Model fits in FP16? (70B needs ~140GB → NO for single A100)
│   │   └─ → FP16, no quantization flag needed
│   │
│   └─ Model too large for FP16?
│       └─ → AWQ INT4  [PREFERRED over GPTQ on vLLM]
│            5–15% higher throughput than GPTQ (optimized CUDA kernels)
│            Quality: ~2–3% drop on reasoning benchmarks
│            vllm serve <model-awq> --quantization awq
│
└─ A100 (40GB) or consumer GPU (4090)
    └─ → GPTQ or GGUF (broader model availability)
         vllm serve <model-gptq> --quantization gptq

Why AWQ beats GPTQ on vLLM: AWQ (Activation-aware Weight Quantization) identifies weight channels that matter most by analyzing activation magnitudes on a calibration dataset, then protects those channels during 4-bit compression. vLLM ships optimized CUDA kernels specifically for AWQ that deliver 5–15% higher throughput than GPTQ kernels. If you're choosing a quantized checkpoint to download, choose AWQ.

FP8 on H100 — why it's not a trade-off: INT4 quantizes weights only. FP8 quantizes both weights and activations. The H100's Transformer Engine handles FP8 natively in hardware. Because FP8 covers the dynamic range of transformer activations well, quality degradation is near-zero — unlike INT4 where reasoning tasks show measurable regression. FP8 is the right default for H100 in 2026.

Multi-GPU Parallelism — Tensor vs Pipeline

Once a model exceeds a single GPU's VRAM, you need parallelism. vLLM supports two strategies with fundamentally different trade-offs.

Tensor parallelism (--tensor-parallel-size N) splits each transformer layer across N GPUs. All GPUs compute every layer simultaneously, communicating via AllReduce after each attention and FFN block.

Pipeline parallelism (--pipeline-parallel-size N) assigns complete layers to different GPUs. GPU 0 runs layers 1–40, GPU 1 runs layers 41–80. Less communication per step, but GPU 0 sits idle while GPU 1 processes its stage.

{/* DIAGRAM: Parallelism topology
    TENSOR PARALLELISM (tp=2, within one node):
    ┌──────────────────────────────────────────────┐
    │  Layer 1  [GPU0: heads 0-31] [GPU1: heads 32-63]  │
    │                   ↕ AllReduce (NVLink: 600 GB/s)  │
    │  Layer 2  [GPU0: heads 0-31] [GPU1: heads 32-63]  │
    └──────────────────────────────────────────────┘
    All GPUs active every layer. Fast on NVLink, slow on PCIe.

    PIPELINE PARALLELISM (pp=2, across nodes):
    ┌──────────────────────┐    ┌──────────────────────┐
    │  Node 0              │    │  Node 1              │
    │  GPU0: Layers 1–40   │───▶│  GPU1: Layers 41–80  │
    └──────────────────────┘    └──────────────────────┘
    Pass activations at boundaries. Less bandwidth. Pipeline bubbles.
*/}

Decision rule:

• Within a single node with NVLink → tensor parallelism
• Across nodes connected by InfiniBand or Ethernet → pipeline parallelism
• Mixing both: use --tensor-parallel-size within node, --pipeline-parallel-size across nodes

Critical constraint: Pipeline parallelism is not composable with speculative decoding as of vLLM ≤ 0.15.0. If you need speculative decoding for latency-sensitive workloads, you must use tensor parallelism or single-GPU deployment.

Speculative Decoding — 2–5× Latency Gain for the Right Workload

LLM decoding is memory-bandwidth bound at low batch sizes — the GPU reads model weights for each token but does minimal compute. Speculative decoding exploits this by using a small draft model to propose several candidate tokens, which the large target model then verifies in one parallel forward pass.

Without speculative decoding (Llama 3.1 70B, 1 user):
  Token 1: [load 70B weights] → generate → 42ms
  Token 2: [load 70B weights] → generate → 42ms
  Token 3: [load 70B weights] → generate → 42ms
  ...100 tokens = 4,200ms

With speculative decoding (3B draft → 70B verify, k=5):
  Draft:  [load 3B weights] → propose tokens [t1,t2,t3,t4,t5] → 6ms
  Verify: [load 70B weights] → check all 5 in parallel → 45ms
  Accept: 3 of 5 accepted (typical acceptance rate 70–80%)
  ...100 tokens ≈ 3×fewer 70B passes → ~1,400–1,800ms

The gains apply when: concurrency is low (1–4 users), you care about latency over throughput, and draft and target models share the same tokenizer family. At high concurrency (32+ concurrent requests), the GPU is compute-saturated and speculative decoding's overhead costs more than it saves.

# Enable speculative decoding — draft must share tokenizer with target
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --speculative-model meta-llama/Llama-3.2-3B-Instruct \
    --num-speculative-tokens 5 \
    --tensor-parallel-size 2
    # pipeline-parallel-size NOT supported here

Production Configuration Reference

Four complete configs covering the most common deployment scenarios.

Scenario A: Single H100, 8B model, high-throughput API

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --quantization fp8 \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.90 \
    --host 0.0.0.0 --port 8000
# Expected throughput: 8,000–10,000 tokens/sec at high concurrency

Scenario B: 2× H100, 70B model, balanced latency/throughput

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 2 \
    --quantization fp8 \
    --max-model-len 8192 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.92 \
    --host 0.0.0.0 --port 8000
# FP16 baseline: ~3,300 tokens/sec (H100 SXM5)
# FP8 expected:  ~5,000–5,500 tokens/sec

Scenario C: Single A100 80GB, 70B model, memory-constrained

# 70B FP16 = ~140 GB VRAM → won't fit one A100
# AWQ INT4 compresses to ~35 GB → fits with KV cache headroom
vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \
    --quantization awq \
    --max-model-len 4096 \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.90
# Expected: 1,200–1,500 tokens/sec

Scenario D: Low-concurrency latency-critical (internal tool, ≤4 users)

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-model meta-llama/Llama-3.2-1B-Instruct \
    --num-speculative-tokens 5 \
    --quantization fp8 \
    --max-num-seqs 4
# Expected: 2–3× TTFT improvement vs non-speculative baseline

Benchmark Before You Deploy

Never move a vLLM config to production without benchmarking against your actual request distribution. vLLM ships a benchmark script designed for this:

python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --num-prompts 1000 \
    --request-rate 10 \
    --dataset-name sharegpt   # or provide your own request traces

# Four metrics to record before deploying:
# 1. Throughput (output tokens/sec)
# 2. TTFT p50/p95/p99       — time to first token
# 3. TPOT p50/p95           — time per output token (decoding speed)
# 4. Request success rate    — OOM failures show up here first

Run three passes: your baseline FP16 config, your candidate quantized config, and --max-num-seqs at 1.5× your expected peak concurrency. The last pass surfaces OOM failures before they happen in production.

vLLM vs TGI vs Ollama

| | vLLM | TGI (HuggingFace) | Ollama | |---|---|---|---| | Throughput | Highest — PagedAttention + continuous batching | High — continuous batching, no PagedAttention | Low — single-user design | | Multi-GPU | Tensor + pipeline parallelism | Tensor parallelism | Limited | | Quantization | AWQ, GPTQ, FP8, GGUF | AWQ, GPTQ, bitsandbytes | GGUF natively | | Speculative decoding | Yes (tp only, not pp) | Yes | No | | Production maturity | High — Anyscale, Replicate, enterprise standard | High — HuggingFace's own serving stack | Not production-oriented | | Use when | Serving real traffic, multi-GPU, API endpoints | Deep HuggingFace ecosystem, TGI-specific features | Local development, laptop testing only |

The summary: vLLM is the production default for self-hosted LLM serving in 2026. Use TGI if you need bitsandbytes quantization or are deeply integrated with HuggingFace's serving stack. Use Ollama for local development only — its throughput at batch sizes above 4 is not competitive with either.

The one scenario where neither wins: if you need to serve hundreds of fine-tuned LoRA adapters on the same base model concurrently, look at vLLM's --enable-lora mode or purpose-built multi-LoRA servers (S-LoRA, Punica) before designing your infrastructure.