PILLAR · LLM SYSTEMS

LLM inference

vLLM internals, FlashAttention, PagedAttention, speculative decoding, quantization, disaggregated serving. The OpenAI / vLLM / Together / Anthropic loops probe deeply here — know throughput vs TTFT tradeoffs cold and be ready to design a serving stack on the whiteboard.

Read ~40 min Asked at OpenAI, Anthropic, vLLM, Together, Baseten, NVIDIA Difficulty Sr Staff bar

What you'll learn

  1. The decode bottleneck — why inference is memory-bandwidth-bound
  2. KV cache — size formulas, real-world numbers
  3. PagedAttention & vLLM — the OS-VM analogy that fixed inference
  4. Continuous batching — the per-iteration scheduler
  5. Speculative decoding — exact, fast, mathematically clean
  6. Quantization for inference — INT4 / FP8 / SmoothQuant / GGUF
  7. FlashAttention — IO-aware attention v1 → v2 → v3
  8. Parallelism in inference — TP, PP, EP for serving
  9. Sampling — greedy through nucleus and beyond
  10. The latency vocabulary — TTFT, ITL, TPS, throughput
  11. Disaggregated serving — Splitwise, DistServe, Mooncake
  12. Prefix & KV-cache compression — beyond MLA
  13. Inference engines compared — vLLM, SGLang, TRT-LLM, TGI
01
FOUNDATIONS · MENTAL MODEL

The decode bottleneck — why inference is memory-bandwidth-bound

🎯LLM inference has two completely different physics: prefill burns compute, decode burns memory bandwidth — and almost every optimization in this field flows from that single fact.

Before any further optimization is meaningful, you need to understand why LLM serving is hard. It's not one problem — it's two distinct problems sharing the same GPU. Prefill (processing the prompt) behaves like a normal deep-learning workload: compute-bound, benefits from big matrices. Decode (generating each new token one at a time) is something stranger: the GPU spends most of its time reading memory, not computing. This chapter builds the mental model that makes every later optimization make sense.

Two phases — a tale of two bottlenecks

Every LLM inference request goes through exactly two phases:

Prefill
Process the entire prompt in one big forward pass. If the prompt is N tokens, attention costs O(N²) and the FFN costs O(N). You get a lot of compute done per byte of memory read. This is compute-bound — the GPU's arithmetic units are the bottleneck.
Decode
Generate tokens one at a time. Each step: read all model weights from HBM, compute attention for just one new query vector against the cached keys and values, produce one output token. Tiny amount of math per enormous amount of memory reading. This is memory-bandwidth-bound.
Arithmetic intensity — the one number that explains everything

Plain words: Arithmetic intensity is the ratio of floating-point operations done to bytes of memory traffic caused. A high ratio = compute-bound. A low ratio = bandwidth-bound.

Concrete example: Suppose you have a 7B parameter model in BF16 (2 bytes/parameter = 14 GB of weights). Each decode step needs exactly those weights read once, plus the KV cache. Let's count:

  • Weights read: ~14 GB
  • FLOPs done: ~14 billion multiply-adds = ~28 GFLOPs
  • Arithmetic intensity: 28 GFLOPs ÷ 14 GB ≈ 2 FLOPs/byte

An A100 can do ~312 TFLOPS of BF16 compute but only ~2 TB/s of HBM bandwidth. That ratio is 312/2 = 156 FLOPs/byte — the GPU's "roofline crossover point." At 2 FLOPs/byte, a single decode stream is over 75× below the compute ceiling. The memory bus is fully saturated long before the cores are.

Formal definition:

$$I = \frac{\text{FLOPs}}{\text{bytes of HBM traffic}}$$
I = arithmetic intensity; FLOPs = floating-point operations in one forward step; bytes = all data read from/written to high-bandwidth memory

Why this matters: If you don't understand arithmetic intensity, you'll propose the wrong optimization. More FLOPs (better algorithm) only helps when you're compute-bound. Memory compression, caching, and bandwidth tricks only help when you're bandwidth-bound. Decode is bandwidth-bound. That's the sentence to tattoo on your brain.

Why batching is the #1 throughput lever

The problem without batching: For a single decode stream, you read 14 GB of weights to produce one token. Those weights are used for one set of Q/K/V projections and one FFN pass. The GPU's memory bus was maxed out just doing that one read. But the arithmetic units were barely touched.

What batching does: If you batch 32 requests together, you still read the weights once (~14 GB), but now you do 32 sets of Q/K/V projections simultaneously. You've just done 32× the useful work for the same memory cost.

  • 1 request: 14 GB read → 1 token out → intensity ≈ 2 FLOP/byte
  • 32 requests: 14 GB read → 32 tokens out → intensity ≈ 64 FLOP/byte (closer to roofline)

This is why serving systems obsess over batch size. Every doubling of batch size roughly doubles throughput — until you run out of KV cache memory. The tension between "big batch for throughput" and "bounded KV cache for memory" defines the engineering of every serving system in this chapter sequence.

⚠ Clears up: "Decode is slow because LLMs are big"

The naive intuition is that decode is slow because the model has billions of parameters. But a model with the same number of parameters would be fast if all requests were prefills (batched matmuls). The slowness is specifically caused by one-token-at-a-time autoregression — each decode step can only produce one token before needing the previous token's output as input. This makes the effective batch size for compute very small and intensity very low.

⚠ Clears up: "Prefill is the same as training"

Prefill looks like a training forward pass — yes. But in serving you have no gradients, and the KV cache computed in prefill is kept alive for all subsequent decode steps. The memory layout challenge for serving is that multiple requests' caches must coexist in GPU memory with very different lifetimes and lengths — that's what PagedAttention (Ch. 3) solves.

📐 If you get this question — the rule

Trigger: "Why is LLM inference slow?" or "What's the bottleneck in LLM serving?"

  1. Split into two phases immediately: prefill vs decode.
  2. State the bound for each: prefill = compute-bound, decode = memory-bandwidth-bound.
  3. Name arithmetic intensity as the metric that separates them.
  4. Give a concrete number: decode on a 70B model reads ~140 GB of weights per token, giving ~1-2 FLOP/byte vs the GPU's 100+ FLOP/byte crossover point.
  5. Then explain why batching helps decode: amortizes the weight read across N requests.

Never: say "it's slow because LLMs are big" without immediately distinguishing the two phases and explaining the bandwidth math.

Roofline model showing prefill in compute-bound region and single-stream decode in memory-bandwidth-bound region; batched decode moves right toward the compute roof.
TL;DR

Prefill is compute-bound (you process the whole prompt in one big matmul). Decode is memory-bandwidth-bound (every step reads all weights + the KV cache to emit one token). Batching helps decode by amortizing the weight read across requests. Almost every inference optimization follows from this one fact.

✓ Remember
  • Prefill = compute-bound (O(N²) attention, big batched matmul). Decode = memory-bandwidth-bound (one new Q vector, all weights read every step).
  • Arithmetic intensity = FLOPs / bytes. Decode ≈ 1-2 FLOP/byte. GPU crossover ≈ 100+ FLOP/byte. Decode is deeply bandwidth-bound.
  • Batching is the #1 throughput lever: amortizes the enormous weight-read across many requests simultaneously.
  • Everything in this page — KV cache, PagedAttention, speculative decoding, quantization, disaggregation — is a direct response to this single memory-bandwidth bottleneck.
Tricky interview questions — chapter 01
Q1. What is arithmetic intensity and why does it matter for LLM serving?
Arithmetic intensity is FLOPs performed per byte of HBM memory traffic. It determines whether a workload is compute-bound (high intensity, arithmetic units bottleneck) or memory-bandwidth-bound (low intensity, memory bus bottleneck). Decode has intensity ≈ 1-2 FLOP/byte for a single stream; the GPU's roofline crossover is ~100+ FLOP/byte. So decode is 50-100× below the compute ceiling — the memory bus saturates before the cores get any real work. Every optimization targeting decode latency or throughput must improve the memory access pattern or amortize it.
Q2. Why is decode memory-bandwidth-bound but prefill is compute-bound?
Prefill processes N tokens simultaneously in one big batched matrix multiplication. With N=1000 tokens, the attention computation does O(N²) = 1M operations per head; the matrix is big enough that the arithmetic units are the constraint. Decode generates one token at a time: one new query vector attends over the cached K,V. That's O(L) per head (L = current sequence length) — tiny compute, but you still must read all model weights (~14-140 GB depending on model size). The ratio is inverted: lots of bytes moved, almost no FLOPs done.
Q3. If you're running decode on a 70B model in BF16 (2 bytes), what is the approximate arithmetic intensity per token for a single request?
70B parameters × 2 bytes = ~140 GB weight read per decode step. Per-step FLOPs ≈ 2 × 70B = 140 GFLOPs (two FLOPs per parameter in a linear layer). Intensity = 140 GFLOPs ÷ 140 GB = 1 FLOP/byte. An H100 SXM has ~3.35 TB/s HBM bandwidth and ~990 TFLOPS BF16 — crossover at ~990/3.35 ≈ 295 FLOP/byte. At 1 FLOP/byte, a single decode stream uses less than 0.4% of compute capacity. This is why GPU utilization looks terrible on decode-heavy workloads with small batch sizes.
Q4. How does batching improve decode throughput? What limits it?
In decode, the weights must be read once per step regardless. With batch size B, the same weight read now produces B tokens instead of 1 — effective intensity scales linearly with B. Going from batch=1 to batch=32 raises intensity from ~1 to ~32 FLOP/byte, roughly 32× better throughput (until other bottlenecks kick in). The limit is KV cache memory: each request in the batch holds a KV cache proportional to its sequence length. Fitting 32 long-context requests simultaneously requires 32× the KV cache memory, which can easily exceed GPU HBM. The tension between large batches (for throughput) and bounded KV cache (for memory) is the central engineering problem of LLM serving.
Q5. A colleague says "just use a faster GPU with more TFLOPS to speed up decode." Why is this mostly wrong?
Decode is memory-bandwidth-bound, not compute-bound. Doubling TFLOPS on a GPU that is bandwidth-bound does almost nothing — the bottleneck is the HBM read speed. What you need is more HBM bandwidth (e.g., H100's 3.35 TB/s vs A100's 2 TB/s gives a meaningful speedup; Blackwell HBM3e goes further). You also need techniques that reduce the bytes read: quantization (W4 vs BF16 halves the weight bytes read), KV cache compression, and batching (amortizes the read). Simply upgrading TFLOPS is the wrong axis for decode.
Q6. At what batch size does decode "cross the roofline" on an H100?
H100 SXM BF16: ~990 TFLOPS compute, ~3.35 TB/s HBM bandwidth → crossover at ~295 FLOP/byte. For a 70B model (intensity at batch=1 is ~1 FLOP/byte), you'd need batch ≈ 295 to reach the compute roof. In practice, KV cache memory and scheduling constraints cap effective batch size well below this for long-context workloads. For shorter contexts (e.g., 1k tokens), KV cache per request is small enough that large batches become feasible. This is why short-context, high-concurrency workloads benefit enormously from large batch sizes.
Q7. How does speculative decoding interact with the memory-bandwidth bottleneck?
Speculative decoding accelerates decode by having a small draft model propose K tokens, then verifying all K in one target-model forward pass. The key: that one verification pass reads the full target model weights once but produces (on average) more than one accepted token. This effectively increases the arithmetic intensity of each target-model memory read — instead of 1 token per weight-read, you get α×K tokens (α = acceptance rate). It's a way to amortize the enormous weight-read cost, similar conceptually to batching, but across time rather than across requests.
Q8. What is the difference between "throughput" and "latency" in the context of LLM decode?
Latency (specifically inter-token latency, ITL) is how long each individual decode step takes — this is bounded by memory bandwidth for a single stream. Throughput is total tokens produced per second across all requests — this scales with batch size. The two are in tension: increasing batch size improves aggregate throughput but worsens per-user ITL because each decode step now has more work to do (more Q vectors, larger KV cache reads). A serving system must set a target on one (e.g., ITL ≤ 50ms P95) while maximizing the other.
Q9. Why does a transformer's FFN contribute more to the bandwidth bottleneck than attention during decode?
During decode, the attention computation is over the cached K,V for the sequence — the KV cache is stored in HBM and read, but attention compute per token is O(L×d_head) per head. The FFN involves two large matrices (W₁ and W₂), each of size d_model × d_ff (e.g., 4096×16384 for a 70B model). These matrices must be fully read for every token in every layer. For a 70B model, FFN weights dominate the per-token weight-read. This is why KV cache compression (Ch. 2) and MQA/GQA (which reduce KV sizes) help less than weight quantization for raw decode throughput.
Q10. What happens to arithmetic intensity during prefill as the prompt length increases?
As prompt length N increases, prefill FLOPs scale as O(N²) (attention) + O(N) (FFN). Memory reads scale as O(N) for the KV computation plus the fixed model weights. The ratio grows with N — longer prompts are more compute-bound. For short prompts (N=100), prefill may still be somewhat bandwidth-limited. For N=10,000+, it becomes heavily compute-bound. This is why disaggregating prefill and decode onto different GPU types can make sense: prefill wants high-FLOP-per-byte GPUs; decode wants high-bandwidth GPUs.
02
FOUNDATIONS · MEMORY

KV cache — size formulas, real-world numbers

🎯The KV cache is the reason LLMs can generate at all without re-running the whole model from scratch — and it's also the reason a single long-context request can consume as much GPU memory as the model itself.

Autoregressive generation requires attending over every previously generated token at each new step. Without caching, that means re-running all attention computations from scratch for each new token — O(L²) total work for a sequence of length L. The KV cache avoids this by storing the key and value tensors from prior steps. But that storage costs real memory, and its size can dwarf the model weights for long contexts. This chapter gives you the formulas and the real numbers every inference engineer must know cold.

Why the KV cache exists — the failure without it

Without a KV cache: To generate token 1000, you'd need to re-run all 1000 tokens through every attention layer, computing K and V for tokens 1-999 from scratch each time. For 80 layers, that's 80 full attention passes over 999 tokens just to get one new token. Total work for L tokens ≈ O(L²×d) — quadratic in sequence length.

With a KV cache: Tokens 1-999 already computed their K and V during earlier steps. Store them. For token 1000, only compute Q, K, V for the new token, then attend Q_{1000} over all cached K_{1..999}. Work per step: O(L×d) instead of O(L²×d). The cache trades memory for compute — every time.

Worked example (4-token sequence, 1 head, d_head=4):

Step 1 (token "The"): compute K_1=[0.1, 0.2, 0.3, 0.4], V_1=[...]. Cache: {1: (K_1, V_1)}
Step 2 (token "cat"): compute K_2, V_2. Attend Q_2 over K_1, K_2. Cache: {1,2: ...}
Step 3 (token "sat"): compute K_3, V_3. Attend Q_3 over K_1..K_3. Cache: {1,2,3: ...}
Without cache at step 3: recompute K_1, K_2, K_3 from embeddings again — wasted.
The size formula — derive it, don't memorize it

For a single token at a single transformer layer, the cache stores:

  • One K vector: n_kv_heads × d_head floats
  • One V vector: n_kv_heads × d_head floats
  • Total: 2 × n_kv_heads × d_head floats

Across all layers and all tokens in a sequence:

$$\text{KV cache} = n_\text{layers} \times L \times 2 \times n_\text{kv\_heads} \times d_\text{head} \times n_\text{bytes}$$
n_layers = transformer depth; L = sequence length (tokens); 2 = K and V; n_kv_heads = number of KV heads (same as Q heads for MHA, fewer for GQA/MQA); d_head = dimension per head; n_bytes = bytes per element (2 for BF16/FP16)

The "2" in front accounts for K and V — a common mistake is to forget it.

Worked example — Llama 3 70B at 128k context

This is the canonical whiteboard number. Llama 3 70B architecture:

n_layers
80
n_kv_heads (GQA)
8 (not 64 — GQA with 8 KV heads shared across 64 Q heads)
d_head
128
dtype
BF16 = 2 bytes
context length
128k = 131,072 tokens

Step-by-step derivation:

  • Per token, per layer: 2 × 8 × 128 × 2 = 4,096 bytes = 4 KB
  • Per token, all 80 layers: 4,096 × 80 = 327,680 bytes ≈ 320 KB
  • At 128k context: 320 KB × 131,072 ≈ 40 GB per request

The counterfactual (why GQA matters): If Llama 3 70B used full MHA (64 KV heads instead of 8), the KV cache would be 8× larger:

  • Per token per layer: 2 × 64 × 128 × 2 = 32,768 bytes = 32 KB
  • At 128k context: 32 KB × 80 layers × 131,072 ≈ 320 GB per request

An H100 SXM has 80 GB HBM. A single MHA 70B request at 128k context would need 4× the GPU's entire memory — completely unservable. This is precisely why GQA was invented.

With MLA (DeepSeek-style): Latent dim ~512 instead of n_kv_heads × d_head = 8×128 = 1024:

  • Per token per layer: 2 × 512 × 2 = 2,048 bytes = 2 KB
  • At 128k: 2 KB × 80 × 131,072 ≈ ~20 GB (half of GQA)
  • With further compression (d_latent≈128): ~3 GB — more than 10× smaller than GQA 40 GB
MHA vs GQA vs MQA vs MLA — the KV head evolution
ArchitectureKV headsPer-token-per-layer bytes (BF16)Quality vs MHAExample
MHAn_heads (= n_q_heads)2 × n_heads × d_head × 2BaselineGPT-2, early LLaMA
MQA12 × d_head × 2~5% degradationPaLM, early Falcon
GQA(g)g (8 typical)2 × g × d_head × 2~matches MHALlama 2/3, Mistral, Gemma
MLA—2 × d_latent × 2~matches MHADeepSeek-V2/V3

GQA intuition: Instead of each query head having its own unique K and V, group the Q heads into g groups; all Q heads in a group share one K and one V. This is lossless-ish: adjacent Q heads attend to similar keys anyway. g=8 is 8× cheaper than MHA; quality matches within 0.5%.

MLA intuition (DeepSeek): Don't store K and V at all — store a low-rank latent vector c per token. At inference, project c back up to K and V on the fly. The latent vector is much smaller than n_kv_heads × d_head, so the cache is tiny.

⚠ Clears up: "GQA reduces model quality"

GQA and MQA are architectural choices made at training time. A model trained with GQA from scratch (like Llama 2 and 3) achieves near-identical quality to an equivalent MHA model. The "quality loss" figures cited for MQA (~5%) come from papers that converted MHA models post-hoc via up-training. Designing for GQA from the start has essentially no quality cost at modern scales.

📐 If you get this question — the rule

Trigger: "How much memory does the KV cache take for model X at context Y?"

  1. Write the formula: n_layers × L × 2 × n_kv_heads × d_head × n_bytes.
  2. Identify n_kv_heads carefully — it's NOT n_q_heads for GQA/MQA models. For Llama 3 70B, it's 8, not 64.
  3. Compute per-token-per-layer first, then multiply up. For Llama 3 70B GQA: 2×8×128×2 = 4096 bytes = 4 KB per token per layer.
  4. State the Llama 3 70B at 128k benchmark: ~40 GB per request (GQA). MHA equivalent would be ~320 GB.
  5. Conclude: this is why GQA/MLA exist, and why KV cache is the dominant memory cost during long-context serving.

Never: confuse n_kv_heads with n_q_heads — that's an 8× error in the final answer for Llama 3 70B.

TL;DR

Per-token-per-layer KV memory is 2 · n_kv_heads · d_head · n_bytes. For Llama 3 70B at 128k context with GQA: ~40 GB per request. With MHA it would be ~320 GB per request — totally unservable, which is exactly why GQA / MLA exist. KV cache is the dominant memory cost during long-context inference and the dominant bandwidth cost during decode.

✓ Remember
  • Formula: n_layers × L × 2 × n_kv_heads × d_head × n_bytes. The 2 is K + V. n_kv_heads ≠ n_q_heads in GQA/MQA.
  • Llama 3 70B at 128k with GQA (8 KV heads) = ~40 GB per request. MHA (64 heads) = ~320 GB — 4× a full H100.
  • MLA (DeepSeek) stores a latent vector instead of K/V directly. At d_latent≈128: ~3 GB at 128k — more than 10× better than GQA.
  • Quality order: MHA ≥ GQA ≥ MQA, but GQA (trained from scratch) matches MHA at 8× lower KV cost.
Tricky interview questions — chapter 02
Q1. What is the KV cache and why is it necessary for autoregressive generation?
The KV cache stores the key (K) and value (V) tensors computed for each token in each attention layer during prior generation steps. Without it, producing token L requires recomputing K and V for all L-1 prior tokens from scratch — O(L²) total work. With the cache, each new step only computes K, V for the one new token and attends over the stored cache — O(L) per step. The cache trades memory for compute: each request keeps KV proportional to its sequence length alive in GPU HBM for the duration of generation.
Q2. Derive the KV cache size formula from scratch.
In MHA, each token at each layer produces one K vector (n_kv_heads × d_head floats) and one V vector (n_kv_heads × d_head floats). That's 2 × n_kv_heads × d_head floats per token per layer. For a sequence of L tokens and n_layers transformer blocks, total floats = n_layers × L × 2 × n_kv_heads × d_head. Multiply by bytes per float (2 for BF16, 4 for FP32) to get bytes. For GQA/MQA, n_kv_heads is smaller than n_q_heads — that's the only change.
Q3. Work out the Llama 3 70B KV cache at 128k context step by step.
Llama 3 70B: 80 layers, 8 KV heads (GQA), d_head=128, BF16 (2 bytes). Per-token-per-layer: 2 × 8 × 128 × 2 = 4,096 bytes. Per-token (all 80 layers): 4,096 × 80 = 327,680 bytes ≈ 320 KB. At 131,072 tokens (128k): 320 KB × 131,072 ≈ 40 GB. The model weights themselves are ~140 GB in BF16, so the KV cache at 128k is about 28% of the weight size per request — and unlike weights, multiple concurrent requests each need their own KV cache.
Q4. What would the KV cache be if Llama 3 70B had used MHA instead of GQA?
With MHA, n_kv_heads = n_q_heads = 64 instead of 8 (an 8× increase). Per-token-per-layer: 2 × 64 × 128 × 2 = 32,768 bytes. Per-token: 32,768 × 80 = 2.56 MB. At 128k tokens: 2.56 MB × 131,072 ≈ 320 GB per request. An H100 SXM only has 80 GB HBM total. The model weights alone are ~140 GB across two GPUs. A single request at 128k would need 4× the entire single-GPU HBM just for KV cache — it could never be served on fewer than 5-6 H100s per request. This is why GQA is not optional for long-context models.
Q5. Explain GQA: what is the group structure and why does quality hold?
In Grouped Query Attention, the n_q_heads query heads are partitioned into g groups of (n_q_heads / g) heads each. Each group shares one K head and one V head. So n_kv_heads = g. At inference, each Q head in a group attends over the same K and V as its group partners. Quality holds because adjacent Q heads in a standard MHA model tend to attend to similar key regions anyway — the grouping doesn't lose much useful diversity. The empirical result: GQA-8 (g=8) on a 70B model matches MHA quality within <0.5% on most benchmarks when trained from scratch.
Q6. What is MLA (Multi-head Latent Attention) and how does it achieve ~10× KV reduction?
MLA (introduced in DeepSeek-V2) replaces the per-token K and V cache with a single low-rank latent vector c per token per layer. The K and V matrices are parameterized as low-rank projections: K = W_K × c, V = W_V × c. At inference, only c is cached (d_latent floats per token per layer), and K/V are recomputed from c on the fly. If d_latent=512 vs d_model=8192 (n_heads × d_head), the cache is 16× smaller. DeepSeek-V2 reports quality matching MHA at ~7% of the KV cache size. The tradeoff: a small recompute cost per attention call to project c → K, V.
Q7. Why is the KV cache the dominant memory cost during long-context serving, not the model weights?
Model weights are a fixed cost shared across all requests in a batch. A single copy of Llama 3 70B at BF16 = ~140 GB total, spread across GPUs via TP. But every concurrent request has its own KV cache proportional to its sequence length. At 128k context, one request needs ~40 GB (GQA). Ten concurrent requests need 400 GB of KV cache alone — vastly exceeding the shared 140 GB weight cost. For long-context serving, the KV cache grows linearly with both concurrency and context length, and it's the binding memory constraint. The model weights are constant; the KV cache is variable and unbounded.
Q8. How does the KV cache connect to the memory-bandwidth bottleneck of decode?
During decode, each step reads: (a) all model weights for the layer computations, and (b) the entire KV cache for that request to perform attention. For a 128k-context request on Llama 3 70B GQA, that's ~40 GB of KV cache read in addition to ~140 GB of weights — total ~180 GB of HBM traffic per decode step. This is the "dominant bandwidth cost during decode" in the TL;DR. With batch=1, the GPU's arithmetic units do almost nothing while the memory bus is saturated. Large KV caches make the bandwidth problem worse per step and also limit how many requests can be batched (since KV caches compete for HBM space with each other and the weights).
Q9. A request needs 60 GB of KV cache and your GPU only has 80 GB total (with the model using 30 GB via INT4). How do you serve it?
With 80 GB HBM, 30 GB to INT4 weights, 50 GB available for KV. You're short by 10 GB. Options: (1) KV cache quantization: INT8 KV halves cache size to 30 GB — fits; INT4 (KIVI-style) to 15 GB — very comfortable. (2) KV eviction (H2O) — evict infrequently attended tokens, accept ~5% quality drop. (3) Offload part of the KV cache to CPU DRAM (if latency allows). (4) Prefix caching — if the first portion of the request is a cached system prompt, that KV may already be stored and shared. (5) Use disaggregated serving with a separate KV store tier (Mooncake-style). In practice, a combination of INT8 KV quantization + prefix caching is the standard first response.
Q10. Why does the "2" in the formula specifically stand for K and V — couldn't we drop one?
Both K and V are necessary for attention. K is used for the dot-product similarity computation (Q·Kᵀ → attention scores). V is used for the weighted sum (attention scores · V → output). You cannot drop either without losing the attention mechanism entirely. The Q vectors are NOT cached because they're only needed at the current step — you compute Q from the current hidden state and immediately use it; there's no need to revisit prior Q values. So the cache is precisely K and V for all prior tokens, never Q.
03
SYSTEMS · MEMORY MGMT

PagedAttention & vLLM — the OS-VM analogy that fixed inference

🎯Before PagedAttention, serving systems wasted up to 80% of GPU memory on fragmentation — PagedAttention fixed this by applying a 50-year-old OS trick (virtual memory paging) to the KV cache.

Once you understand that the KV cache is the central memory resource of LLM serving (Ch. 2), the next question is: how do you manage it efficiently? The naive approach turns out to be catastrophically wasteful. PagedAttention (Kwon et al., 2023) solved this with a beautiful analogy to operating-system virtual memory — and it's the most impactful single inference paper since the transformer itself. This chapter explains the problem, the solution, and why the OS analogy is exact, not approximate.

The fragmentation disaster — why naive allocation fails

Naive approach: When a request arrives, allocate a contiguous block of KV cache memory sized for the maximum possible sequence length (e.g., 4096 tokens). Keep it alive until the request finishes.

The three waste modes:

Internal fragmentation
Most requests don't use 4096 tokens. A request generating 200 tokens wastes 3896 slots of reserved memory — tied up until it finishes. With 1000 concurrent requests, you're wasting hundreds of GB.
External fragmentation
Different requests finish at different times, leaving "holes" of freed memory between live allocations. You can't fit a new request into two 500-token holes even if you need 600 tokens total, because the holes aren't adjacent.
Reservation waste
You don't know how long a response will be when it starts (generation is autoregressive). So you must reserve the worst case. But p95 output length is often 5-10× shorter than the maximum — you've reserved for the 95th percentile always, even when serving the 5th percentile.

Real-world impact: Benchmarks before PagedAttention showed 60-80% of GPU memory wasted on fragmentation. A system that could theoretically serve 40 concurrent requests was actually serving 8-15.

The OS-VM analogy — blocks, block tables, virtual addresses

Operating systems solved exactly this problem 50 years ago with virtual memory paging. Here's the direct analogy:

OS Virtual MemoryPagedAttention
Physical RAM divided into fixed-size pagesGPU HBM KV cache divided into fixed-size blocks (e.g., 16 tokens each)
Each process has a virtual address spaceEach request has a logical block table
Page table maps virtual → physical pagesBlock table maps logical block slots → physical KV blocks
OS allocates pages on demandSystem allocates KV blocks on demand as new tokens arrive
Multiple processes share physical pages (CoW)Multiple requests share physical KV blocks (prefix sharing)

Worked example — 3 requests, block size 4:

Physical blocks: [B0][B1][B2][B3][B4][B5][B6][B7]  (8 blocks total)

Request A (7 tokens): logical [0,1] → physical [B0,B3]   (non-contiguous!)
Request B (3 tokens): logical [0]   → physical [B1]
Request C (5 tokens): logical [0,1] → physical [B2,B5]

As C grows to 9 tokens: allocate new physical block on demand → logical [2] → [B4]
When B finishes: free B1 immediately → available for any new request

No request needs to own a contiguous stretch of memory. Any free block anywhere can be assigned to any request. Fragmentation drops to near zero.

Prefix sharing — free throughput via copy-on-write

The block table indirection enables something the naive approach cannot do at all: multiple requests sharing the same physical KV blocks for common prefixes.

Concrete example — system prompt sharing:

System prompt: "You are a helpful assistant. Always cite your sources." (12 tokens)
  → KV computed once, stored in physical blocks [B0, B1, B2]

Request 1: "What is the capital of France?"
  → logical [0,1,2] → physical [B0, B1, B2] (shared)
  → logical [3] → physical [B6] (request-specific new tokens)

Request 2: "Explain quantum entanglement."
  → logical [0,1,2] → physical [B0, B1, B2] (same shared blocks!)
  → logical [3] → physical [B7] (different request-specific block)

Two requests, only one set of system-prompt KV blocks in HBM. Throughput improvement for chat workloads with shared system prompts: often 2-5× in token throughput for the prefix portion.

Copy-on-write semantics: If a request needs to modify a shared block (rare, e.g., beam search updating the same prefix with different branches), PagedAttention copies the block before writing — exactly like OS CoW fork semantics.

Block size tradeoff — 16 vs 128
Small blocks (16 — vLLM default)
Low internal fragmentation (at most 15 wasted slots per sequence end). More granular sharing (can share prefixes at 16-token resolution). More block-table entries to manage (more metadata).
Large blocks (128)
Less block-table overhead. Fewer GPU memory operations per attention call (contiguous reads within a block are faster). More wasted space at sequence ends (up to 127 slots).

16 is the vLLM default. For workloads with very predictable, long sequence lengths, larger blocks can improve throughput via better memory locality. Benchmark before changing.

⚠ Clears up: "PagedAttention changes the attention math"

The attention computation is mathematically identical whether KV blocks are contiguous or scattered. The PagedAttention CUDA kernel simply follows the block table to gather the scattered physical blocks into a logical sequence before computing attention scores. The output is bit-identical to what you'd get from a contiguous implementation. PagedAttention is a memory management scheme, not a new attention algorithm.

◆ Interview probe

"Why did the vLLM paper (Kwon 2023, arxiv 2309.06180) show such dramatic throughput improvements — up to 24× over naive HuggingFace serving?" Answer: The baseline wasted 60-80% of GPU memory on fragmentation, limiting concurrency. PagedAttention nearly eliminated fragmentation, allowing 3-5× more concurrent requests. Combined with continuous batching (Ch. 4), each GPU cycle is now filled with useful work instead of waiting for the oversized, wasteful KV allocations to drain.

📐 If you get this question — the rule

Trigger: "How does vLLM manage memory?" or "What is PagedAttention?"

  1. State the problem: naive KV allocation is contiguous and worst-case-sized → 60-80% fragmentation waste.
  2. Name the OS analogy immediately: "PagedAttention is OS virtual memory paging applied to the KV cache."
  3. Explain the two components: fixed-size physical KV blocks + per-request logical block table mapping logical → physical.
  4. Give the key benefits: near-zero fragmentation, dynamic growth (allocate blocks as tokens arrive), prefix sharing across requests (copy-on-write).
  5. Cite the paper: Kwon 2023, arxiv 2309.06180.

Never: say PagedAttention changes the attention algorithm — it's a memory manager. The math is identical; the CUDA kernel just follows a block table.

Diagram showing physical KV blocks scattered in GPU HBM, multiple requests' logical block tables pointing to non-contiguous physical blocks, and two requests sharing the same prefix blocks.
✓ Remember
  • PagedAttention = OS virtual memory for the KV cache. Fixed blocks + per-request block tables. Kwon 2023.
  • Eliminates internal and external fragmentation → allows 3-5× more concurrent requests per GPU.
  • Block table indirection enables prefix sharing: multiple requests share the same physical KV blocks for common prefixes (system prompts, few-shot examples) via copy-on-write.
  • Block size 16 (default) trades slightly more metadata for lower fragmentation and finer-grain sharing.
  • This is the most impactful LLM inference paper of 2023, enabling vLLM's 2-24× throughput improvement over naive serving.
Tricky interview questions — chapter 03
Q1. What is the memory waste problem PagedAttention solves?
Naive KV cache allocation reserves a contiguous block of GPU memory sized for the maximum possible sequence length for each request. Since most requests are much shorter, the reserved-but-unused slots waste memory. Additionally, as requests finish, they leave non-contiguous holes that can't be compacted (GPU memory isn't easily defragmented while other requests are running). Empirically, 60-80% of KV cache HBM was wasted before PagedAttention, limiting concurrency by 3-5×.
Q2. Explain the block table mechanism in PagedAttention.
The KV cache HBM is divided into fixed-size physical blocks (e.g., 16 tokens × n_kv_heads × d_head floats each). Each request maintains a logical block table — an array where entry i points to the physical block holding its i-th logical block of KV. When a request generates new tokens, the system allocates the next free physical block (from anywhere in HBM) and appends it to the block table. The PagedAttention CUDA kernel follows the block table to gather the scattered physical blocks into a logical sequence when computing attention. No contiguous allocation is ever needed.
Q3. How does prefix sharing work in PagedAttention, and what's the memory benefit?
If two requests share a common prefix (e.g., the same system prompt), their block tables for the prefix portion can point to the same physical KV blocks. Only one copy of the prefix KV exists in HBM, shared by reference-count. When a request diverges (starts generating request-specific tokens), a new physical block is allocated; if it needs to modify a shared block, it copies first (copy-on-write). For a chatbot where every request starts with a 500-token system prompt, prefix sharing means the system prompt's KV is computed once and served to all concurrent requests for free — potentially halving or better the effective KV memory usage.
Q4. Walk through the vLLM scheduler loop for one decode step.
The vLLM scheduler runs before every forward pass: (1) Check the free-block pool. (2) For each running request, compute how many new blocks it will need (usually 0 or 1). (3) If free blocks are available for all running requests, proceed to decode. (4) If a running request can't get a block, preempt it (swap its KV to CPU DRAM or free its blocks and recompute later). (5) Fill remaining free capacity by admitting waiting requests from the queue (they start with a prefill pass). (6) Execute the batched forward pass for all admitted+running requests. After the pass, new KV blocks are written and block tables updated.
Q5. What is the OS virtual memory analogy for each component of PagedAttention?
Physical RAM pages → physical KV blocks in HBM. Virtual address space per process → logical block space per request. Page table (virtual → physical) → block table (logical block index → physical block pointer). Page fault + allocation → on-demand new physical block when a request generates a token that would fill the current block. Copy-on-write fork semantics → copy-on-write for shared KV blocks when a request needs to modify a prefix shared with another request. The analogy is not approximate — it is exact in structure.
Q6. Why is block size 16 the vLLM default rather than 1 or 256?
Block size 1 eliminates fragmentation entirely but creates enormous block-table overhead (one pointer per token) and many small GPU memory operations. Block size 256 reduces metadata but wastes up to 255 slots at the end of each request and forces coarser prefix-sharing (can only share at 256-token granularity). Block size 16 is a practical Goldilocks point: <15 wasted tokens at sequence end (< 6% waste for a 256-token response), reasonable block-table size (~n_blocks pointers per request), fine-grained prefix sharing at 16-token resolution. The vLLM authors chose it empirically; other frameworks use 32 or 64 with similar justification.
Q7. How does PagedAttention enable beam search to be efficient?
Beam search maintains k beams (candidate sequences) simultaneously. With naive contiguous allocation, each beam needs its own full KV buffer — k× the memory. With block tables, beams sharing a common prefix (every step before they diverge) can share physical KV blocks. When a beam diverges, only the new blocks are unique; all prefix blocks are shared. Copy-on-write semantics handle the divergence. For a 4-beam search with 1000 shared tokens and 100 unique: without PagedAttention, 4× 1100-token buffers = 4400 slots. With PagedAttention: 1000 shared + 4×100 unique = 1400 slots — 3× savings.
Q8. What happens when the KV block pool is exhausted — how does vLLM handle memory pressure?
When free blocks run out, vLLM has two options: (1) Preemption-swap: pause a running request, write its KV blocks to CPU DRAM, free those GPU blocks for other requests. When the request is resumed, copy KV back to GPU (expensive — PCIe bandwidth limited). (2) Preemption-recompute: discard the KV blocks and restart the request from scratch (replaying the prompt prefill) when space becomes available. Swap is better when TTFT is critical; recompute is better when KV blocks are large and PCIe bandwidth would be too slow. vLLM supports both, controlled by the scheduler policy.
Q9. RadixAttention in SGLang extends PagedAttention — what does it add?
SGLang's RadixAttention organizes the prefix cache as a radix trie (prefix tree) over KV blocks. When a new request arrives, the tree is searched for the longest matching prefix — in O(log n) time where n is the number of cached prefixes. This enables aggressive automatic prefix sharing not just for a single shared system prompt but for any shared prefix structure: multi-turn conversations (each turn is a prefix of the next), few-shot prompting, chain-of-thought prefixes, etc. Compared to PagedAttention's simpler hash-based prefix matching, the trie finds longer matches in hierarchical conversations and structured outputs.
Q10. If a 100-token request and a 10,000-token request are both running with block size 16, how many blocks does each hold?
100 tokens: ceil(100/16) = 7 blocks (with 12 unused slots in the last block — 12/16 = 75% internal fragmentation in that block, but only 12/100 ≈ 12% overall). 10,000 tokens: ceil(10000/16) = 625 blocks (with 0 unused — 10000 is divisible by 16). The short request wastes 12 slots of 16 in its final block. The long request wastes nothing. PagedAttention's fragmentation is bounded by (block_size - 1) ≈ 15 slots per sequence — regardless of how many thousands of tokens the request generates. This is the "bounded waste" property that makes it so much better than pre-allocated worst-case buffers.
04
SYSTEMS · SCHEDULING

Continuous batching — the per-iteration scheduler

TL;DR

Static batching collects N requests, runs all to completion, ships the batch. Short requests sit and wait for long ones — terrible utilization. Continuous batching (Orca, Yu 2022) reschedules per iteration: finished requests leave, new ones join. vLLM, TGI, TensorRT-LLM all do this. It's the second-most-important scheduling idea after PagedAttention.

Static vs continuous

Static batching

  • Collect N requests, run all to completion.
  • Short requests wait for long ones.
  • Throughput limited by tail of batch.
  • Good for offline / batch jobs only.

Continuous batching (Orca)

  • Scheduler operates per-iteration.
  • After each forward pass, finished requests leave; new ones join.
  • Maximizes GPU utilization.
  • Standard in vLLM, TGI, TRT-LLM.
PITFALL — head-of-line blocking from prefill
Even with continuous batching, a long prefill (say, a 100k-token prompt) freezes the decode stream for everyone in the batch — one user's TTFT becomes everyone else's TTFT. Fix: chunked prefill (Sarathi-Serve, Ch. 11) or full prefill/decode disaggregation.
REMEMBER
  • Continuous = per-iteration scheduling; finished out, new in.
  • Static batching is fine for offline; never for serving.
  • Long prefills still cause head-of-line blocking — chunk them.
05
DECODING · ACCELERATION

Speculative decoding — exact, fast, mathematically clean

TL;DR

A small, fast draft model proposes K tokens. The big target model verifies all K in parallel with one forward pass. Accept tokens up to the first rejection; sample one fresh token from a residual distribution at the rejection point. The math is constructed so the marginal output distribution exactly equals p_target. ~2-3× speedup when draft acceptance is high and draft cost is low.

The algorithm

(Leviathan 2023, arxiv 2211.17192.) A small draft model generates K tokens; the target model verifies all K in parallel with one forward pass; accept tokens up to first rejection, sample one more from corrected distribution. Always exact (samples from target distribution).

THE INSIGHT — speculative decoding accept/reject math

Why the output distribution is exactly p_target

For each drafted token t:

  • Accept with probability min(1, p_target(t) / p_draft(t)).
  • If rejected, sample one fresh token from the residual distribution: 
    p_residual(t') = max(0, p_target(t') − p_draft(t')) / Z
    (truncated to non-negative, normalized). Stop processing further drafted tokens.
  • If all K drafts accepted, sample one bonus token from p_target.

Why exact? Condition on accepted vs rejected and integrate. The marginal distribution of each emitted token is:

p_emit(t) = p_draft(t) · min(1, p_t/p_d)              [accepted path]
          + (1 − Σ p_draft · accept) · p_residual(t)   [rejected path]
        = p_target(t)                                  [the algebra works out]

Frontier-lab probe: "Prove speculative decoding samples from p_target." Be ready to write this on a whiteboard.

Speedup ~2–3× depending on draft acceptance rate. Best when draft is fast (~5% of target cost) and aligned (high acceptance).

EAGLE / EAGLE-2/3

(Li 2024, arxiv 2401.15077.) Uses target model's hidden states (not just tokens) as input to a small auto-regressive head that drafts. Higher acceptance because the draft sees richer context.

Medusa

(Cai 2024.) Multiple parallel decoding heads on the target model; each head predicts token at position +1, +2, +3, etc. Tree-based verification. No separate draft model. Simpler deployment.

Lookahead decoding

(Fu 2024.) Generates n-grams via Jacobi iteration on the target model itself; verifies through a unified attention pattern. Draft-free.

PITFALL — speculative decoding tradeoff
Wins only when (a) draft acceptance rate is high (similar distribution to target) AND (b) draft is fast (≤ ~5% of target cost). Loses if draft is too divergent (low acceptance, wasted compute) or too slow (overhead dominates). Tune both.
REMEMBER
  • Speculative decoding is exact — output distribution = p_target by construction.
  • Accept prob = min(1, p_t/p_d); on reject, sample from (p_t − p_d)_+ / Z.
  • EAGLE leads on acceptance rate; Medusa is simpler; Lookahead is draft-free.
06
SYSTEMS · LOW-PRECISION

Quantization for inference — INT4 / FP8 / SmoothQuant / GGUF

TL;DR

The 2026 production sweet spot: W4A16 (INT4 weights via GPTQ/AWQ, BF16 activations) for cost-sensitive serving; FP8 W8A8 on H100/Blackwell for max throughput. SmoothQuant migrates outliers from activations to weights so W8A8 actually works. NF4 is for QLoRA. GGUF is llama.cpp's format for CPU/edge.

INT8 / FP8 — the W8A8 family

Per-channel weight quantization; activations either quantized too (W8A8) or kept in FP16 (W8A16). LLM.int8() (Dettmers) handles outliers via mixed precision. FP8 is native in H100 — generally W8A8 with per-tensor or per-channel scales.

SmoothQuant (Xiao 2023, arxiv 2211.10438): activations have outlier channels that ruin quantization; migrate the difficulty from activations to weights via a diagonal scale s:

(X · diag(s)⁻¹) · (diag(s) · W)

Now activations have smaller outliers and weights absorb the scale. This is the standard recipe for production W8A8 to actually work.

W4A16 — most common production deployment

INT4 weights (GPTQ/AWQ), bf16 activations. Smaller than W8A8 in storage but equal compute (matmul still runs in bf16). Pure W4A4 is much harder due to activation outliers.

NF4 (QLoRA)

(Dettmers, arxiv 2305.14314.) "NormalFloat" — quantization levels chosen to match a normal distribution's quantiles. Good for weights (which are roughly Gaussian).

Microscaling (MX)

Block-wise scales — better outlier handling. MXFP8 / MXFP6 / MXFP4. Hardware support in Blackwell.

GGUF

llama.cpp's K-quants (Q4_K_M, Q5_K_S, etc.) — block-wise non-uniform schemes good for CPU/edge inference. Designed for the GGUF container format.

EXAMPLE — 2026 sweet spot

Weights INT4 (GPTQ/AWQ) or FP4 (Blackwell), activations FP8 or BF16. Tensor cores actually accelerate INT8 / FP8 matmul, so W8A8 has both storage and compute wins on H100/Blackwell. W4A16 wins on memory bandwidth (smaller weight read) but uses BF16 tensor cores.

REMEMBER
  • W4A16 = most common production. FP8 W8A8 = max throughput on H100+.
  • SmoothQuant is the trick that makes W8A8 actually work in production.
  • GPTQ uses a Hessian; AWQ protects salient channels.
07
SYSTEMS · KERNELS

FlashAttention — IO-aware attention, v1 → v2 → v3

TL;DR

Standard softmax-attention materializes an L×L matrix in HBM — IO-bound at long sequence length. FlashAttention tiles the computation and uses online softmax to keep partial state in SRAM. v1 (Dao 2022) showed ~3× speedup. v2 improved work partitioning. v3 (Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao 2024) exploits H100 features (TMA, warp specialization, FP8) to hit ~75% of H100 peak.

FA1 — the IO-aware idea

(Dao 2022, arxiv 2205.14135.) Computes attention in tiles. Standard softmax requires the full row to normalize. Online softmax: maintain running max and sum, rescale as new tiles arrive. Avoids materializing the L×L attention matrix in HBM. IO-aware; ~3× faster for typical seq lengths, much more for long.

FA2 — better partitioning

(Dao 2023, arxiv 2307.08691.) Improved work partitioning, reduced non-matmul FLOPs, parallelizes across sequence dim. Better GPU occupancy.

FA3 — H100-specific

FA3 (Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao 2024, arxiv 2407.08608). Note: FA3 lead author is Jay Shah, not Tri Dao (who is on the paper). Frontier-lab interviewers actually probe the attribution.

EXAMPLE — what FA3 actually exploits on H100
  • TMA (Tensor Memory Accelerator) — async memory copy from HBM → SMEM, freeing the warp scheduler.
  • Warp specialization — producer/consumer warps overlap data movement and compute.
  • FP8 with two-stage scaling — preserves precision through the matmul.

Result: up to ~75% of H100 peak (~740 TFLOPS BF16).

REMEMBER
  • FA = tile + online softmax → never materialize the L×L matrix.
  • FA3 lead author = Jay Shah; Tri Dao is on the paper. Get the attribution right.
  • FA3 features: TMA, warp specialization, FP8 two-stage scaling.
08
SYSTEMS · PARALLELISM

Parallelism in inference — TP, PP, EP for serving

TL;DR

Tensor Parallel shards weight matrices across GPUs (within NVLink, two all-reduces per layer, TP ≤ 8). Pipeline Parallel shards layers (lower BW, pipeline bubbles). Expert Parallel places different MoE experts on different GPUs (all-to-all dispatch). Serving combines them — DeepSeek V3 ran EP across 64-256 GPUs with custom DualPipe overlap kernels.

Tensor Parallel (TP)

Shard each weight matrix across GPUs. For attention: shard heads. For FFN: shard hidden dim. Two all-reduces per layer (after attention output proj, after FFN W₂). Bandwidth-hungry → keep within NVLink domain (TP ≤ 8 typically).

Pipeline Parallel (PP)

Shard layers across GPUs. Lower bandwidth; introduces pipeline bubbles. Mostly used for very large models when TP within node isn't enough.

Expert Parallel (EP)

For MoE, place different experts on different GPUs. Token dispatch via all-to-all. Combines with TP and DP. DeepSeek V3 used EP across 64-256 GPUs with custom comm/compute overlap kernels (DualPipe).

PITFALL — TP across PCIe
TP requires all-reduce twice per layer. Across PCIe (no NVLink), you'll spend more time in collectives than compute. Cap TP at the NVLink domain — usually 8 GPUs in a single node — and use PP or replicas across nodes.
REMEMBER
  • TP within NVLink (≤ 8). PP across nodes if needed. EP for MoE.
  • Two all-reduces per layer in TP — bandwidth is the bottleneck.
  • DualPipe (DeepSeek) overlaps comm and compute across pipeline stages.
09
DECODING · SAMPLING

Sampling — greedy through nucleus and beyond

TL;DR

Greedy is deterministic and prone to repetition. Top-k clips to a fixed count; top-p (nucleus) clips to a cumulative probability mass; min-p adapts to distribution flatness. Beam search is great for translation, terrible for creative gen. Repetition / frequency / presence penalties are hacky but ubiquitous in production.

The full menu

REMEMBER
  • Top-p with p ≈ 0.9-0.95 + temperature ≈ 0.7-1.0 = the safe production default.
  • Min-p is an under-rated upgrade: adapts to distribution flatness.
  • Beam search for translation; never for chat.
10
SYSTEMS · METRICS

The latency vocabulary — TTFT, ITL, TPS, throughput

TL;DR

TTFT = how long until the first token streams (prefill-bound, FLOP-heavy). ITL = per-token decode time (memory-bandwidth-bound). TPS per stream = 1/ITL. Aggregate throughput = sum across streams (higher batch = higher aggregate but worse per-stream ITL). SLA-driven scheduling balances these.

The four numbers everyone confuses

MetricWhat it isWhat it's bound by
TTFT (Time to First Token)Latency from request → first tokenPrefill (compute the prompt's KV in one pass). FLOP-bound for long prompts.
ITL (Inter-Token Latency)Per-token decode timeMemory-bandwidth-bound (must read weights + KV every step).
TPS (Tokens Per Second)Per-stream throughput = 1/ITLSame as ITL.
Aggregate throughputTokens/sec across all concurrent streamsHigher batch → higher aggregate but worse per-stream ITL.

Tradeoff: low TTFT + low ITL = expensive (small batches, lots of GPUs idle). High aggregate throughput = large batches, slower per-user. SLA-driven scheduling balances these.

PITFALL — autoscaling lag
LLM serving has long startup time (load 70-405 GB weights). HPA-style autoscaling on QPS is too slow — by the time new pods come up, the SLA is blown. Provision for the p95, not the median, and use queue-depth-based scaling, not QPS.
REMEMBER
  • TTFT = prefill (compute). ITL = decode (bandwidth). Don't conflate.
  • Higher batch → better aggregate throughput, worse per-stream ITL.
  • Autoscale on queue depth, provision for p95.
11
SYSTEMS · DISAGGREGATION

Disaggregated serving — Splitwise, DistServe, Mooncake

TL;DR

Prefill is compute-bound, decode is memory-bandwidth-bound. Co-locating them causes interference (decode latency spikes when prefill runs). Disaggregate them onto separate GPU pools and transfer KV via NVLink/RDMA. Splitwise (Microsoft 2023), DistServe (Berkeley 2024), Mooncake (Kimi 2024) are the canon. Sarathi-Serve's chunked prefill is the alternative for single-pool deployments.

Why disaggregate

Prefill is compute-bound, decode is memory-bandwidth-bound. Running them on the same GPU causes interference — decode latency spikes during prefill (head-of-line blocking).

Disaggregation: separate prefill and decode pools. Prefill GPUs do bulk compute, then transfer KV cache to decode GPUs over fast interconnect (NVLink, RDMA). Independent scaling.

The canon

DeepSeek's open-source serving stack uses this pattern.

Chunked prefill (Sarathi-Serve) — the single-pool alternative

Sarathi-Serve (Agrawal 2024, arxiv 2403.02310): instead of running long prefills as one big forward pass (which freezes decode for everyone), chunk the prefill into smaller pieces and interleave them with decode iterations. Removes TTFT spikes during sustained traffic. Now standard in vLLM and TensorRT-LLM.

Prefix caching

When many requests share a prompt prefix (system prompts, few-shot examples, RAG context), cache the KV for that prefix and reuse. Hash the prefix tokens, look up in a KV-cache pool. vLLM's PagedAttention enables sharing at block granularity.

Massive throughput wins for chatbots with shared system prompts and for agentic workloads with growing transcripts.

REMEMBER
  • Disaggregate prefill from decode → independent scaling, no interference.
  • Splitwise / DistServe / Mooncake are the canon. Mooncake adds a KV-cache tier.
  • Sarathi-Serve chunks prefill — alternative for single-pool deployments.
  • Prefix caching is free throughput when system prompts are shared.
12
SYSTEMS · KV COMPRESSION

Prefix & KV-cache compression — beyond MLA

TL;DR

Once you've adopted GQA or MLA architecturally, the next lever is on-the-fly KV compression. KIVI quantizes K per-channel and V per-token to INT2 with little quality loss. H2O / Heavy Hitters evict everything but the top-attention tokens. ChunkAttention dedups shared prefix chunks across requests via prefix tree. CLA shares KV across consecutive layers.

The KV-compression toolkit

EXAMPLE — RadixAttention (SGLang)

SGLang's prefix-cache structure: a radix trie over KV cache. When a new request shares prefix with cached requests, the tree walk gives O(log n) lookup of the longest matching prefix. Enables aggressive prefix-cache reuse with low overhead.

REMEMBER
  • KIVI: INT2 KV with per-channel K, per-token V — almost free quality.
  • H2O: keep heavy hitters, evict the rest. ~5% quality cost.
  • ChunkAttention + CLA / MLKV cut cache further. Stack with GQA / MLA.
  • SGLang's RadixAttention is the prefix-cache reference impl.
13
ECOSYSTEM · ENGINES

Inference engines compared — vLLM, SGLang, TRT-LLM, TGI

TL;DR

vLLM is the open-source throughput champion (PagedAttention, continuous batching). SGLang is catching up fast with RadixAttention and great structured-output performance. TensorRT-LLM is the NVIDIA-only speed king but model-conversion friction is real. TGI is the easy-deploy option. llama.cpp / MLC-LLM / ExecuTorch own the edge.

The matrix

EngineStrengthsWeak/Notes
vLLMPagedAttention, continuous batching, broad model support, openThroughput champion in 2024; SGLang catching up
SGLangRadixAttention (prefix tree caching), fast structured generation, very fast tool-call decodingNewer; growing rapidly in 2025
TensorRT-LLMBest-in-class on Nvidia HW (kernel fusion, paged kv, in-flight batching)Nvidia-only, model conversion friction
TGI (Hugging Face)Easy deployment, broad model coverageThroughput trails vLLM/SGLang
llama.cppCPU + Metal + CUDA, GGUF format, edge-friendlyThroughput-limited at scale
MLC-LLM, ExecuTorchOn-device (mobile, browser via WebGPU)Edge / consumer product
REMEMBER
  • vLLM = open default. SGLang = catching up, best for structured output.
  • TRT-LLM = peak Nvidia perf, friction tax to convert.
  • llama.cpp / MLC-LLM / ExecuTorch = edge.

0 → hero reading path for LLM inference

  1. foundation vLLM blog — start with PagedAttention post, then continuous batching, then prefix caching
  2. foundation TGI docs
  3. foundation Sebastian Raschka — Coding the KV Cache from scratch
  4. build Walk through vLLM source — start with vllm/model_executor/layers/attention/
  5. build Implement speculative decoding in numpy — drill until you can do it in 30 min
  6. depth PagedAttention / vLLM paper (Kwon 2023)
  7. depth FlashAttention v1 (Dao 2022)
  8. depth FlashAttention v2
  9. depth FlashAttention v3 (Shah et al.)
  10. depth Speculative Decoding (Leviathan 2023)
  11. depth EAGLE
  12. depth Splitwise — disaggregated serving
  13. depth Mooncake (Kimi)
  14. depth Sarathi-Serve — chunked prefill
  15. depth GPTQ + AWQ + SmoothQuant for quantization
  16. depth LMSYS blog (SGLang authors) — RadixAttention etc.

LLM inference quiz — readiness check

  1. Walk through one decode step inside vLLM.
    Show answer

    Scheduler picks ready requests up to memory limit. For each: compute Q, K, V from previous token's hidden state; write K, V to next free block in the request's block table. PagedAttention kernel: attention over all blocks for that request. FFN. Sample next token. Update block table. New requests joining first do prefill (one big forward over their prompt).

  2. Why is decode memory-bandwidth-bound?
    Show answer

    Each decoded token requires reading all weights (~140 GB for 70B fp16) plus the full KV cache. Compute is small (one new Q vector). Batching helps because you reuse one weight read across many requests in the batch.

  3. How would you reduce TTFT for 100k-token prompts?
    Show answer

    Chunked prefill (Sarathi-Serve), prefix caching, sequence/context parallel for prefill, more GPUs allocated to that request. The bottleneck is FLOPs.

  4. Speculative decoding tradeoff?
    Show answer

    Wins when draft acceptance rate is high (similar distribution to target) and draft is fast (≤ ~5% target cost). Loses if draft too divergent (low acceptance) or too slow (overhead dominates).

  5. Compare GQA vs MQA vs MLA on KV cache size.
    Show answer

    MHA: 2·n_heads·d_head per token per layer. MQA: 2·d_head. GQA(g): 2·g·d_head. MLA: 2·d_latent (much smaller, e.g., 512 vs 8192 for d_model=8192). Quality: MHA ≥ GQA ≥ MQA, MLA matches MHA at 7% cache size.

  6. What's the bottleneck for serving 10k QPS chat?
    Show answer

    Memory bandwidth (decode), KV cache size (concurrent context), GPU count (cost). Mitigations: GQA/MLA, FP8 weights, prefix caching, continuous batching, speculative decoding, model routing (cheap → reasoning).

  7. Design a serving stack for a reasoning model with 8k hidden CoT tokens per request.
    Show answer

    Heavy decode load (8k tokens × users); huge KV cache. Disaggregated prefill (cheap) and decode (expensive); aggressive prefix caching across reasoning segments; queue with priority for premium tier; possibly spec-decoding with a weaker model for early CoT phase.

  8. How does FA3 use H100 features?
    Show answer

    TMA (Tensor Memory Accelerator) for async memory copy from HBM → SMEM. Warp-specialized producer/consumer pattern overlapping data move with compute. FP8 matmuls with two-stage scaling. Up to 75% of H100 peak (~740 TFLOPS bf16).

  9. Prove speculative decoding is exact.
    Show answer

    For drafted token t: accept with prob min(1, p_t/p_d). On reject: sample from residual (p_t − p_d)_+ / Z. Marginal distribution of emitted token = p_d · accept + (1 − p_d · accept) · residual = p_t (algebra works out). Each emitted token is exactly distributed as p_t.

  10. Worked example: KV cache for Llama 3 70B at 128k context?
    Show answer

    GQA: 8 KV heads, d_head=128, 80 layers, bf16. Per token per layer: 2 · 8 · 128 · 2 = 4096 B. Per token: × 80 = ~320 KB. At 128k tokens: ~40 GB per request. MHA equivalent (64 KV heads): 8× = ~320 GB.

  11. Why disaggregate prefill and decode?
    Show answer

    Prefill is compute-bound; decode is memory-bandwidth-bound. Same GPU running both → decode latency spikes during prefill. Disaggregation: separate pools, transfer KV cache via NVLink/RDMA. Independent scaling. Splitwise / DistServe / Mooncake.

  12. Difference between PagedAttention block size 16 vs 128?
    Show answer

    Smaller (16): less internal fragmentation, more granular sharing across requests, more block-table overhead. Larger (128): less metadata, fewer GPU memory operations per attention, more wasted space at sequence ends. 16 is the vLLM default; experimentation needed for specific workloads.

  13. What is RadixAttention (SGLang)?
    Show answer

    Prefix tree (radix trie) over KV cache. When a new request shares prefix with cached requests, the tree walk gives O(log n) lookup of the longest matching prefix. Enables aggressive prefix-cache reuse with low overhead. SGLang's contribution.

  14. Top-p vs top-k vs min-p?
    Show answer

    Top-k: keep top k probabilities. Top-p (nucleus): keep smallest set with cumulative prob ≥ p. Min-p: keep tokens with p ≥ p_threshold · max(p) — adaptive (wider in flat distributions).

  15. EAGLE vs Medusa for speculative decoding?
    Show answer

    EAGLE: small auto-regressive head conditioned on target's hidden states (richer context → high acceptance). Medusa: multiple parallel heads predicting +1, +2, +3 with tree verification. EAGLE-2/3 push further. Medusa simpler deployment; EAGLE higher acceptance.

  16. What is SmoothQuant?
    Show answer

    Migrate quantization difficulty from activations to weights via diagonal scaling: (X · diag(s)−1) · (diag(s) · W). Now activations have smaller outliers, weights absorb the scale. Standard recipe for production W8A8.

  17. What's the difference between W4A16 and W8A8?
    Show answer

    W4A16: INT4 weights, bf16 activations. Smaller storage; matmul still in bf16. Most common production. W8A8: INT8 weights AND activations. Faster matmul (uses int8 tensor cores) but harder due to activation outliers. Need SmoothQuant.

  18. What is chunked prefill (Sarathi-Serve)?
    Show answer

    Instead of running long prefills as one big forward (freezing decode for all users), chunk the prefill and interleave with decode iterations. Removes TTFT spikes during sustained traffic. Now standard in vLLM and TensorRT-LLM.

  19. What is KIVI / H2O?
    Show answer

    KIVI: per-channel quant of K, per-token quant of V → INT2 KV cache with minimal quality loss. H2O (Heavy Hitters): identify the small set of tokens that dominate attention; evict the rest. Drastically smaller cache; ~5% quality drop.

  20. Continuous batching vs static batching — explain.
    Show answer

    Static: collect N requests, run all to completion. Short requests wait for long ones. Continuous: per-iteration scheduling. After each forward pass, finished requests leave; new requests join. Maximizes GPU utilization. vLLM, TGI, TensorRT-LLM all do this.

Next: Distributed training