SYSTEMS · TRAINING INFRASTRUCTURE

Distributed training

DDP fits a 7B model on one node. Past 70B you need ZeRO/FSDP. Past 100B you need 3D parallelism. Past 405B you need failure-handling for hardware that breaks every few hours. This page is the full memory-bandwidth-bubble triangulation a Sr Staff candidate is expected to do live.

Read ~35 min Asked at Anthropic, OpenAI, DeepMind, Meta, xAI Difficulty Sr Staff bar

What you'll learn

The memory budget — why DDP doesn't fit at 70B+
ZeRO & FSDP — sharding optimizer state, grads, params
Tensor parallel (Megatron) — splitting a matmul across GPUs
Pipeline parallel — stages, microbatches, the bubble
Sequence & context parallel — when sequences are too long
Expert parallelism for MoE — and the all-to-all bottleneck
Mixed precision — FP16 / BF16 / FP8 (the H100 era)
Activation checkpointing & gradient accumulation
The collectives toolkit — AllReduce, AllGather, AllToAll
Hardware reality — NVLink, NVSwitch, IB, NCCL, SHARP
Checkpointing strategies — sync, async, sharded
Failures at 10k+ GPU scale — what kills naive runs

FOUNDATIONS · MEMORY ACCOUNTING

The memory budget — why DDP doesn't fit at 70B+

🎯Before you pick a parallelism strategy, count bytes: Adam mixed-precision needs 16 bytes per parameter — that's 1.12 TB for a 70B model, and no GPU on Earth holds that alone.

This chapter establishes the mathematical foundation for every distributed-training decision: how much memory does training actually consume, and why does that number force you into multi-GPU strategies before you can even think about throughput? We walk the 16-bytes-per-parameter accounting from first principles, show the 70B example concretely, and establish the hierarchy of strategies that flows from the memory constraint.

Why memory, not compute, is the binding constraint

The common assumption is that you distribute training to go faster. That's secondary. The primary reason is memory: a model that doesn't fit on one GPU cannot be trained on one GPU, no matter how slow you're willing to be. Throughput is a luxury; fitting is a requirement.

To understand what "fitting" means, you need to know every byte that training consumes. There are three categories:

Model state

Weights, gradients, optimizer slots — the "16 bytes/param" we'll derive.

Activations

Intermediate tensors saved for backward. Proportional to batch size × sequence length × hidden dim. Can be reduced by checkpointing (ch 08).

KV cache (inference only)

Not a training concern, but worth naming so you don't conflate inference and training memory.

This chapter focuses on model state — the part that is fixed regardless of batch size.

The 16 bytes/param breakdown — derived from first principles

Plain-words first: when you train with Adam in mixed precision, you maintain five separate tensors for each scalar parameter. Each tensor has a type (FP32 or BF16) that determines how many bytes it occupies. Sum them up and you get 16.

Tiny example: suppose your model has exactly 1 parameter: a scalar weight w. What does the trainer allocate?

A BF16 copy of w for fast forward/backward (2 bytes).
A BF16 gradient tensor matching w (2 bytes).
An FP32 "master weight" copy — the source of truth for the optimizer update (4 bytes).
An FP32 first-moment (momentum) m for Adam (4 bytes).
An FP32 second-moment (variance) v for Adam (4 bytes).

Total: 2 + 2 + 4 + 4 + 4 = 16 bytes.

State tensor	Dtype	Bytes/param	Purpose
BF16 weights	BF16	2	Used in forward/backward matmuls
BF16 gradients	BF16	2	Output of backward pass
FP32 master weights	FP32	4	Source of truth for optimizer step
FP32 Adam `m` (1st moment)	FP32	4	EMA of gradient — tracks direction
FP32 Adam `v` (2nd moment)	FP32	4	EMA of squared gradient — tracks curvature
Total	—	16	Per parameter, before activations

Why FP32 for optimizer state? The Adam update rule subtracts a tiny number from the master weight. If both are in BF16, weight updates smaller than ~0.01% of the weight's magnitude are rounded to zero — learning stalls. FP32 gives enough precision that even tiny gradient signals accumulate correctly over thousands of steps. This is the ZeRO paper (Rajbhandari et al. 2019, Table 1) accounting.

$$M_\text{state} = 16 \times N_\text{params}$$

$M_\text{state}$: model state memory in bytes. $N_\text{params}$: number of scalar parameters. The 16 factor = 2 (BF16 weights) + 2 (BF16 grads) + 4+4+4 (FP32 master + Adam m + v).

For a 70B model: $70 \times 10^9 \times 16 = 1.12 \times 10^{12}$ bytes = 1.12 TB. An H100 has 80 GB of HBM. A B200 has 192 GB. Neither fits 1.12 TB. Distributed training is not a performance choice — it is a memory requirement.

DDP — the baseline everyone starts from

What it does: Data Parallel Distributed (DDP) places a full copy of the model on each GPU. Training batches are split across GPUs (each GPU sees a different micro-batch). After backward, gradients are all-reduced across all replicas so every replica's optimizer step sees the same gradient. Each replica then independently updates its copy of the weights.

Concrete example: 4 GPUs, batch of 256 samples → each GPU processes 64 samples → each GPU produces its own gradient → all-reduce averages the 4 gradient tensors → each GPU runs Adam → all 4 GPUs now have identical updated weights. Next step.

Memory cost: every GPU holds the full 16 bytes/param. For 70B that's 1.12 TB per GPU. There is no sharding. DDP is replication, not partitioning.

When it's the right answer: when the full model state fits on one GPU with room for activations. Rough rule: sub-7B models in BF16 + moderate batch. Above that, the optimizer state alone overflows a single H100.

What breaks without knowing this: candidates who say "just use DDP on 8 GPUs" for a 70B model look like they haven't done real distributed training. The question "does it fit?" must come before the question "how fast?"

Worked scenario: 70B on 8×H100

Let's walk through the arithmetic step by step so the constraints become concrete.

Model state: 70B × 16 bytes = 1.12 TB total.
Per-GPU budget (8×H100): 8 × 80 GB = 640 GB total HBM across the node.
DDP attempt: each GPU needs the full 1.12 TB. Impossible. DDP is dead at 70B.
ZeRO-3 (state sharded across 8 ranks): per-GPU state = 1.12 TB / 8 = 140 GB. Still doesn't fit — we also need ~20–40 GB for activations at any useful batch size.
ZeRO-3 + activation checkpointing: with selective recomputation (ch 08), activation memory drops to ~10–15 GB. Now per-GPU sits at ~150–155 GB. Still tight — but add the fact that ZeRO-3 streams parameters in and out rather than holding all of them simultaneously, and with a small per-GPU batch, it fits.
Bottom line: at 70B on 8×H100 you need both ZeRO-3 sharding and activation checkpointing just to fit. Throughput is a secondary concern.

⚠ Clears up: "but can't I just use FP16 and cut it in half?"

FP16 weights and grads = 2 bytes each = 4 bytes. But the optimizer state stays in FP32 (12 bytes/param) because Adam needs the precision. So you'd have 4 + 12 = 16 bytes. The total doesn't change. FP8 training (ch 07) can cut the forward/backward bytes further, but FP32 master weights and optimizer state remain — bringing you to ~14 bytes/param minimum with current FP8 recipes.

📐 If you get this question — the rule

Trigger: "How much memory does training a 70B model require?" or "Why can't you train a 70B model on a single GPU?"

State the 16 bytes/param breakdown: 2+2+4+4+4 (BF16 weights, BF16 grads, FP32 master, FP32 Adam m, FP32 Adam v).
Compute: 70B × 16 = 1.12 TB. H100 = 80 GB. "That's 14 H100s just for model state, before a single activation."
Explain the consequence: DDP is replication → doesn't help. You need sharding (ZeRO/FSDP) to split the 16 bytes across ranks.
Segue naturally to ZeRO stages (ch 02).

Never: say "just use fp16, it halves the memory" without acknowledging that optimizer state stays in FP32.

✓ Remember

Adam mixed-precision = 16 bytes/param: 2+2+4+4+4. Memorize the breakdown, not just the number.
70B model = 1.12 TB of optimizer state. H100 = 80 GB. No single GPU holds a 70B model under Adam.
DDP = full replication. It does not reduce per-GPU state. It only parallelizes the batch.
"Will it fit?" is the first question in any training design conversation. Throughput comes second.
Optimizer state stays FP32 for precision — halving weight dtype doesn't halve total memory.

TL;DR

Adam mixed-precision training needs 16 bytes per parameter: BF16 weights (2) + BF16 grads (2) + FP32 master weights (4) + FP32 Adam m (4) + FP32 Adam v (4). A 70B model = 1.12 TB of state, which is 14× the HBM of an H100. DDP replicates state — it can't help. Distributed training is a memory requirement before it is a performance optimization. Every chapter that follows describes a different strategy for spreading those 16 bytes across GPUs.

Tricky interview questions — chapter 01

Q1. A 70B model, Adam optimizer, mixed precision. How much memory does the model state consume? Walk through each component.

16 bytes per parameter × 70×10⁹ = 1.12 TB. The 16 breaks down as: 2 (BF16 forward/backward weights) + 2 (BF16 gradients) + 4 (FP32 master weights for optimizer) + 4 (FP32 Adam first moment m) + 4 (FP32 Adam second moment v). The FP32 components dominate at 12 bytes/param because Adam requires the precision to accumulate small gradient signals without rounding to zero. An H100 has 80 GB; you'd need at least 14 H100s just for model state before considering activations or KV cache.

Q2. Why doesn't DDP solve the 70B memory problem, even with 16 GPUs?

DDP replicates the full model state on every GPU — it doesn't shard. With 16×H100 (16×80 GB = 1.28 TB total), you might think it fits in aggregate, but each individual GPU still holds 1.12 TB — which is 14× its 80 GB budget. DDP parallelizes the batch (data), not the state (model). The right tool is ZeRO-3/FSDP, which shards optimizer state + gradients + parameters across ranks so each rank holds only 1/N of the 16 bytes/param.

Q3. Why is FP32 used for Adam's optimizer state when the model runs in BF16?

Adam updates: w ← w − η·m/√(v+ε). The correction step is often tiny relative to the weight magnitude — on the order of 0.001% or less. BF16 has 7 mantissa bits, giving a relative precision of about 1 in 128 (~0.8%). If the update is smaller than 0.8% of the weight, BF16 rounds it to zero. Over thousands of steps, this causes the optimizer to "freeze" certain parameters. FP32's 23 mantissa bits give ~7× more decimal digits, enough that even small corrections accumulate faithfully.

Q4. Someone proposes training a 7B model on a single A100-80GB. Is that feasible under Adam mixed precision?

Model state: 7B × 16 = 112 GB. That already exceeds the A100's 80 GB — so naive Adam mixed precision does not fit on a single A100. You have options: (a) Use ZeRO-Offload to push optimizer state to CPU RAM — slow but functional. (b) Use a memory-efficient optimizer like 8-bit Adam (bitsandbytes), which quantizes m and v to 8-bit, cutting optimizer state from 8 to 2 bytes/param → total becomes ~6 bytes/param → 42 GB, which fits. (c) Use LoRA to only train a small adapter, reducing trainable parameters drastically.

Q5. A colleague says "just train in FP16 to halve the memory." What's wrong with this reasoning?

Two issues. First, cutting weights and grads from BF16 (2 bytes each) to FP16 doesn't change their size — both are 2-byte formats. The actual question is whether to use a lower-precision format like FP8 for the forward/backward pass. Second, even if you use FP8 (1 byte) for weights and grads, the FP32 optimizer state (12 bytes/param) remains — it cannot be quantized further without correctness problems. So the minimum under current recipes is roughly 1+1+4+4+4 = 14 bytes/param — not 8 or less. DeepSeek V3's FP8 recipe achieves close to this minimum.

Q6. What's the difference between model state memory and activation memory? Which one does ZeRO address?

Model state is the fixed 16 bytes/param: weights, gradients, optimizer slots. It's constant regardless of batch size or sequence length. Activation memory is the intermediate tensors stored during forward for use in backward — it scales with batch size × sequence length × hidden dim × number of layers. ZeRO (Stages 1-3) addresses model state only. Activation memory is addressed separately by activation checkpointing / selective recomputation (ch 08). Both are often needed simultaneously at 70B+.

Q7. At 70B on 8×H100 with ZeRO-3, does the model state now fit per GPU?

ZeRO-3 shards all 16 bytes/param across 8 ranks: per-GPU state = 1.12 TB / 8 = 140 GB per rank. That still exceeds 80 GB! However, ZeRO-3 doesn't hold all shards simultaneously — it streams parameters layer by layer (all-gather before a layer, discard after). The resident memory at any moment is roughly (1 layer's parameters) + (your shard of the full optimizer state). For a 70B model with 80 layers and 8 ranks, this becomes feasible at small batch sizes. Add activation checkpointing to reduce activation memory to ~10 GB and the full recipe fits.

Q8. A 13B model, single GPU, you can only see OOM errors during the optimizer step. Why then, not during the forward pass?

During the forward pass, only BF16 weights (2 bytes/param) are resident plus activations. That's 26 GB for 13B params — well within 80 GB if batch is small. The optimizer step materializes the FP32 master weights + Adam m + Adam v simultaneously: 3 × 4 bytes × 13B = 156 GB, which overflows. The BF16 forward pass fools you into thinking you have plenty of headroom — the OOM only hits when the full 16 bytes/param are all resident at once during the optimizer update. Fix: use ZeRO-1 (shard only optimizer state across DP ranks) or 8-bit Adam.

Q9. How does the memory math change for a SGD optimizer vs Adam?

Plain SGD: no m, no v. State = BF16 weights (2) + BF16 grads (2) + FP32 master weights (4) = 8 bytes/param. SGD with momentum adds one FP32 momentum buffer (4 bytes) → 12 bytes/param. Adam at 16 bytes/param is the 4-bytes-heavier choice. For a 70B model: SGD = 560 GB (still needs 7× H100); Adam = 1.12 TB. Both are infeasible on a single GPU — the difference only matters at the margin when choosing how many GPUs you need.

Q10. You're given a 70B training run that finished successfully on 64 H100s. Estimate the per-GPU peak HBM usage.

Model state (ZeRO-3 sharded over 64): 1.12 TB / 64 = 17.5 GB. Activation memory at a typical batch (e.g., 2048 tokens/GPU with selective recomputation): ~8–15 GB depending on sequence length and hidden dim. Gradient buffers (already counted in model state). Temporary workspace: ~2–4 GB. Total: roughly 30–40 GB peak per H100. This leaves 40–50 GB of slack — consistent with running at moderate batch without OOM errors. If they were pushing batch higher, they'd be using activation checkpointing to trade FLOPs for activation memory.

Q11. Why does the ZeRO paper give exactly 16 bytes, not some other number like 18?

The count is exact: the Adam mixed-precision recipe stores exactly the five tensors in the table. One common mistake is counting the gradient twice (once as BF16, once "communicated" during all-reduce). The BF16 gradient (2 bytes) is written by backward and then all-reduced; there is no separate "communication buffer" counted in the 16. Another mistake is adding "workspace" or "buffer" memory — those are temporary and framework-level, not counted in ZeRO's theoretical accounting. The 16 bytes/param is a theoretical floor; in practice PyTorch adds small overheads for metadata.

Q12. How would the accounting change if you used a lion optimizer instead of Adam?

Lion (Chen et al. 2023) only maintains one momentum term (not two like Adam), and uses sign(m) for the update. State: BF16 weights (2) + BF16 grads (2) + FP32 master weights (4) + FP32 momentum m (4) = 12 bytes/param. For 70B: 12 × 70B = 840 GB — still 10.5× H100, but a 25% reduction vs Adam. This is a meaningful saving at scale. The tradeoff is that Lion tends to need larger batch sizes and lower learning rates to converge, so total token cost may increase.

DATA PARALLEL · SHARDING STATE

ZeRO & FSDP — sharding optimizer state, grads, params

🎯ZeRO's key insight: instead of replicating all 16 bytes/param on every GPU, shard them — and AllReduce = ReduceScatter + AllGather means Stages 1 and 2 cost the same comm as DDP but far less memory.

Chapter 01 established that 70B models need 1.12 TB of optimizer state — far beyond one GPU. This chapter shows the precise mechanism by which ZeRO (Zero Redundancy Optimizer) partitions that 1.12 TB across data-parallel ranks, stage by stage, with the critical insight that Stages 1 and 2 are essentially free relative to DDP. We then cover FSDP as PyTorch's production implementation of ZeRO-3, and explain when to use offloading as a last resort.

The core insight: AllReduce = ReduceScatter + AllGather

Plain words: DDP uses a single AllReduce to average gradients across N GPUs after backward. AllReduce can be decomposed into two operations: a ReduceScatter (each GPU ends up with 1/N of the result), followed by an AllGather (everyone broadcasts their slice to everyone else). These two ops together move the same total bytes as AllReduce.

Why this matters: if you do the ReduceScatter but skip the AllGather, each rank holds only its 1/N slice of the gradient. You've used no extra bandwidth — but you now have a sharded gradient that each rank only needs its own slice of to update its shard of optimizer state. This is the entire foundation of ZeRO.

$$\text{AllReduce}(T) \equiv \text{ReduceScatter}(T) \;+\; \text{AllGather}(T/N)$$

$T$: tensor volume in bytes. The total byte volume of ReduceScatter + AllGather equals that of a ring AllReduce. Decomposing enables opportunistic sharding of the scattered portion.

The three ZeRO stages — what each one shards

From Rajbhandari et al. 2019 (arXiv 1910.02054). ZeRO progressively partitions training state across $N$ DP ranks. The paper's Table 1 gives exact per-rank memory:

Stage	What is sharded	Per-rank state	Comm vs DDP
DDP (baseline)	Nothing — full replication	16 bytes/param	1× (1 AllReduce on grads)
ZeRO-1	Optimizer state only (FP32 m, v, master weights)	4 + 12/N bytes/param	Identical to DDP
ZeRO-2	Optimizer state + gradients	2 + 14/N bytes/param	Identical to DDP
ZeRO-3	Optimizer state + gradients + parameters	16/N bytes/param	~1.5× DDP

Example with N=8, 70B model:

ZeRO-1: per-GPU = 4 + 12/8 = 4 + 1.5 = 5.5 bytes/param → 385 GB. Still doesn't fit (H100=80 GB) alone, but now 8 GPUs' aggregate need is 385 GB (vs 1.12 TB for DDP). Each GPU is responsible for updating 1/8 of the params.
ZeRO-2: per-GPU = 2 + 14/8 = 2 + 1.75 = 3.75 bytes/param → 262 GB aggregate. Free comm win again.
ZeRO-3: per-GPU = 16/8 = 2 bytes/param → 140 GB aggregate. Plus the streaming trick (parameters all-gathered layer by layer, then freed), peak resident is much lower.

Why Stages 1 and 2 are free — the arithmetic

DDP does: AllReduce on full gradient tensor (volume = 2M bytes for BF16, M = number of params).

ZeRO-2 does: ReduceScatter on gradients (volume = M bytes/rank), then each rank independently applies its optimizer step to its gradient shard, then AllGather of parameter updates is deferred to the next layer fetch. The ReduceScatter has the same bandwidth cost as the first half of the AllReduce. The AllGather cost is avoided because each rank only needs its own shard. Net: same communication volume as DDP, but half the gradient memory per rank.

The "free" claim: Stages 1 and 2 require no additional communication beyond what DDP already does. They purely reorganize how the existing comm volume is used. If someone says "ZeRO-2 is more communication than DDP," they're wrong — cite the ZeRO paper Table 5.

ZeRO-3 — the 1.5× comm overhead explained

When parameters themselves are sharded, each rank holds only 1/N of the parameters. To run a forward pass through a layer, the rank needs all the layer's parameters. So it must all-gather them from the other ranks, compute, then free the gathered params (to save memory). On backward, it must all-gather again (parameters were freed), run the backward, then reduce-scatter the gradients (to land each rank's gradient shard).

That's: AllGather (forward) + AllGather (backward) + ReduceScatter (backward) = 3 ops. DDP does: AllReduce = 2 equivalent ops. So ZeRO-3 = 3/2 = 1.5× comm. This is the figure from ZeRO paper Table 5 — often misquoted as "2×" in blogs.

$$\text{ZeRO-3 comm} = \underbrace{\text{AllGather}}_\text{fwd} + \underbrace{\text{AllGather}}_\text{bwd} + \underbrace{\text{ReduceScatter}}_\text{bwd} = 1.5 \times \text{DDP}$$

Each of the three collectives costs $\frac{N-1}{N} \cdot M_\text{layer}$ bytes. DDP's AllReduce = $\frac{2(N-1)}{N} \cdot M$. Ratio = $\frac{3}{2}$.

FSDP — PyTorch's production ZeRO-3

Fully Sharded Data Parallel (FSDP) is PyTorch's built-in implementation of ZeRO-3. Key concepts:

Flat parameters

FSDP concatenates all parameters within a "FSDP unit" (typically a transformer block) into one flat buffer, then shards that buffer across ranks.

All-gather on demand

Before the forward pass of each FSDP unit, FSDP all-gathers the full parameter tensor. After the layer, the non-owned parameters are freed.

Reduce-scatter on backward

After backward, FSDP reduce-scatters the gradients so each rank accumulates its own shard's gradient.

FSDP-2 (PyTorch 2.x)

Per-parameter sharding instead of per-flat-buffer. Cleaner composition with tensor parallelism (ch 03). Now the default in modern PyTorch training stacks.

Practical config: wrap each transformer block as one FSDP unit. Use AUTO_WRAP_POLICY or manual wrapping. Set ShardingStrategy.FULL_SHARD for ZeRO-3 behavior; SHARD_GRAD_OP for ZeRO-2 behavior.

ZeRO-Offload and ZeRO-Infinity — escape hatches

When even ZeRO-3 doesn't fit (e.g., trillion-parameter models, or you only have 1–2 GPUs):

ZeRO-Offload

Optimizer state + parts of gradient live in CPU RAM. Optimizer step runs on CPU. GPU only holds BF16 weights + grads. Enables training a 13B model on a single 24 GB GPU. Slow: bounded by CPU-GPU PCIe bandwidth (~32 GB/s vs NVLink's 900 GB/s).

ZeRO-Infinity

Extends Offload to NVMe SSDs. Even slower — NVMe sequential read is ~7 GB/s, 100× slower than NVLink. Used as an absolute last resort; throughput often under 10% of what GPU-resident training achieves. Not a production recipe for serious training runs.

When to mention these in interviews: only as options for researchers who can't get more GPUs, or for inference of very large models where latency is not critical. Don't propose Infinity as a primary training strategy.

⚠ Clears up: "ZeRO-3 shards the model — doesn't that mean inference is impossible?"

ZeRO-3 is a training-time strategy. At inference time, you all-gather the full model on each device (or use separate inference serving strategies). For inference of large models, FSDP is typically converted back to a standard distributed model via fsdp.consolidate_shard_weights(), or you use a separate inference serving system (like vLLM) that does its own tensor parallelism.

📐 If you get this question — the rule

Trigger: "DDP vs ZeRO-3 / FSDP — when would you use each?"

Start with the memory test: does the full 16 bytes/param + peak activation fit on one GPU? If yes → DDP (simplest comm, 1 AllReduce/step).
If state overflows one GPU → ZeRO-1 first (free comm savings). If still not enough → ZeRO-2 (also free). If still not enough → ZeRO-3/FSDP (1.5× comm, but mandatory for 70B+).
Mention that ZeRO-1 and ZeRO-2 are "free" — same comm as DDP. Only ZeRO-3 costs extra comm (1.5×, not 2×).
State the NVLink requirement: ZeRO-3's per-layer AllGathers are bandwidth-hungry; ideally run within an NVLink domain or at least within a node.

Never: say ZeRO-3 is 2× DDP comm (it's 1.5×); or say ZeRO-2 has more comm than DDP (it doesn't).

✓ Remember

ZeRO-1 shards optimizer state (12 bytes/param); ZeRO-2 adds gradients (2 more); ZeRO-3 adds parameters (everything). Each stage ÷N.
ZeRO-1 and ZeRO-2 have identical comm cost to DDP — truly free memory savings.
ZeRO-3/FSDP costs 1.5× DDP comm (2 AllGathers + 1 ReduceScatter vs 1 AllReduce). Not 2×.
FSDP-2's per-parameter sharding is the modern default; it composes with TP cleanly.
Offload and Infinity are last resorts — PCIe and NVMe bandwidth bottleneck them severely.

TL;DR

ZeRO shards the 16 bytes/param across DP ranks in three stages. Stages 1 and 2 (optimizer state, then gradients) cost the same bandwidth as DDP — they're free memory wins. Stage 3 adds parameter sharding, enabling per-rank state of 16/N bytes/param, but costs 1.5× DDP comm because parameters must be all-gathered per layer in both forward and backward. FSDP is PyTorch's production ZeRO-3 implementation. Offload/Infinity are slow fallbacks, not primary strategies.

Tricky interview questions — chapter 02

Q1. What are the three ZeRO stages and what does each one shard?

ZeRO-1: shards optimizer state (FP32 master weights + Adam m + v = 12 bytes/param) across N ranks. Each rank holds 1/N of the optimizer tensors plus a full copy of BF16 weights and grads (4 bytes/param fixed). Per-rank: 4 + 12/N. ZeRO-2: additionally shards gradients (2 bytes/param). Per-rank: 2 + 14/N. ZeRO-3: additionally shards parameters (BF16 weights, 2 bytes/param). Per-rank: 16/N. Only ZeRO-3 requires the parameter all-gather pattern.

Q2. Explain why ZeRO-1 has the same communication cost as DDP.

DDP: after backward, AllReduce the gradient tensor (volume = 2 bytes × N_params). ZeRO-1: after backward, ReduceScatter the gradients (each rank gets 1/N of the result, volume = 2M bytes total, same as DDP). Each rank then applies its optimizer step to its shard, updates its shard of the FP32 master weights, and writes back BF16 weights. The AllGather of the updated BF16 weights is needed before the next forward — this has the same volume as the AllGather half of the AllReduce DDP would have done anyway. Same bytes moved, different timing. ZeRO-1 is "free."

Q3. Why does ZeRO-3 cost 1.5× DDP comm rather than 3× or 2×?

ZeRO-3 performs: (1) AllGather of layer params before forward (volume = M_layer per rank, across N ranks); (2) AllGather of layer params before backward (same volume — params were freed after forward); (3) ReduceScatter of gradients after backward (same volume). Total = 3 × (N-1)/N × M per layer. DDP's AllReduce = 2 × (N-1)/N × M. Ratio = 3/2 = 1.5×. The "3×" misconception comes from counting each op as a full AllReduce; the "2×" misconception comes from comparing AllGather+ReduceScatter to AllReduce without noting they're equivalent to 1× each.

Q4. You have a 13B model and 2×A100-80GB. Which ZeRO stage do you use?

Model state: 13B × 16 = 208 GB total. With N=2 ranks: ZeRO-1 → 4 + 12/2 = 10 bytes/param → 130 GB per rank. Still exceeds 80 GB. ZeRO-2 → 2 + 14/2 = 9 bytes/param → 117 GB. Still exceeds 80 GB. ZeRO-3 → 16/2 = 8 bytes/param → 104 GB. Still exceeds 80 GB (before activations!). Conclusion: even ZeRO-3 across 2 A100s doesn't fit a 13B model at training time. You'd need either more GPUs (4+ A100s → ZeRO-3 gives 52 GB/rank + activations manageable), or ZeRO-Offload (optimizer to CPU), or 8-bit Adam to cut optimizer state.

Q5. FSDP vs DDP — which would you choose for fine-tuning a 7B model on 4×A100-40GB?

7B model state = 7B × 16 = 112 GB. A single A100-40GB has 40 GB. DDP requires 112 GB per GPU — impossible. FSDP (ZeRO-3) with N=4: 112/4 = 28 GB per rank for model state. Plus activations: at a small batch (e.g., 512 tokens × 4 sequences), maybe 5–10 GB. Total ~33–38 GB per rank — fits in 40 GB with careful batch sizing and activation checkpointing. FSDP is the right choice. If you're fine-tuning with LoRA (only training adapters), the trainable parameter count drops dramatically and DDP becomes feasible again.

Q6. What is FSDP-2 and why does it matter more than FSDP-1?

FSDP-1 (PyTorch 1.x/2.0) shards a flat concatenation of all parameters in an FSDP unit — one big flat buffer per unit. This makes it hard to compose with tensor parallelism (which needs individual parameter views) and causes memory fragmentation. FSDP-2 (PyTorch 2.1+) shards at per-parameter granularity. Each parameter is individually sharded, so TP can access its column/row slices cleanly. FSDP-2 also changes the gradient accumulation contract, making it more memory-efficient for irregular shapes. In 2025 practice, FSDP-2 is the default in Torchtitan and other frontier training stacks.

Q7. A colleague says "ZeRO-3 shards the model, so we can train any size model with enough GPUs." What's the caveat?

Three caveats: (1) ZeRO-3's 1.5× comm overhead means that as N grows, you're spending more and more of your bandwidth budget on AllGathers — eventually comm time dominates compute time and adding more GPUs doesn't help. (2) Activation memory is NOT sharded by ZeRO-3 — it scales with batch size and must be addressed separately via activation checkpointing or sequence parallelism. (3) The AllGather pattern is bandwidth-hungry — at very high N, you need NVLink-class bandwidth or the per-layer AllGathers become the bottleneck. At some point, tensor parallelism or pipeline parallelism is a better fit than just scaling ZeRO-3.

Q8. ZeRO-Offload is used to train a 70B model on a single A100. What is the practical throughput hit?

With ZeRO-Offload, optimizer state (12 bytes/param × 70B = 840 GB) lives in CPU RAM. The optimizer step runs on CPU. The GPU-CPU PCIe bandwidth is ~32 GB/s (PCIe 4.0 × 16). Each optimizer step must read and write ~840 GB of optimizer state + gradients over PCIe: 840 GB / 32 GB/s ≈ 26 seconds per step, even ignoring CPU compute. A full GPU-resident run of 70B might do an optimizer step in <1 second on 8 GPUs. Throughput hit is 10–50×. ZeRO-Offload on a single GPU for 70B is a research exercise, not a production recipe. Use it only if you have zero alternatives.

Q9. Explain the memory and communication trade-off between ZeRO-2 and ZeRO-3 with N=64 GPUs for a 70B model.

ZeRO-2: per-rank = 2 + 14/64 = 2 + 0.22 = 2.22 bytes/param × 70B = 155 GB. Doesn't fit (>80 GB). Comm = DDP (1×). ZeRO-3: per-rank = 16/64 = 0.25 bytes/param × 70B = 17.5 GB (sharded state only, with layer-by-layer streaming). Fits with plenty of room. Comm = 1.5× DDP. With 64 GPUs and NVLink topology, the 1.5× comm penalty is generally acceptable (the AllGathers happen in microseconds). Conclusion: at N=64, ZeRO-3 is necessary and worthwhile; ZeRO-2 doesn't even solve the problem.

Q10. What does "per-rank" memory mean vs "total memory across all ranks" in ZeRO context?

ZeRO reporting is always per-rank (per-GPU) — the number you care about for OOM. Total memory across all ranks = per-rank × N. For ZeRO-3 with N=8 and 70B model: per-rank = 16/8 = 2 bytes/param → 140 GB per rank. Total across 8 GPUs = 140 × 8 = 1.12 TB — same as the baseline, just split across 8 machines. ZeRO doesn't reduce total memory; it reduces per-rank memory by eliminating redundancy. DDP's "total" is also 1.12 TB but all concentrated in each single GPU. This is the key distinction: redundancy elimination, not data compression.

MODEL PARALLEL · TENSOR SPLIT

Tensor parallel (Megatron) — splitting a matmul across GPUs

TL;DR

Shard each weight matrix across GPUs so a single matmul becomes a parallel matmul + all-reduce. Megatron's column-then-row pattern needs exactly one all-reduce per direction per MLP block. Bandwidth-hungry — keep TP within a single NVLink domain.

The column-then-row trick

From Shoeybi et al. 2019 (arxiv 1909.08053). For a two-matmul block Z = GeLU(X · A) · B, choose the shard axes so the GeLU is local:

Column-parallel A: each GPU holds A_i (a column slice). Computes Y_i = GeLU(X · A_i). No comm — GeLU is elementwise.
Row-parallel B: each GPU holds B_i (a row slice). Computes partial Z_i. All-reduce (sum) to combine.

Net cost: one all-reduce in forward, one in backward, per MLP block. For attention, shard heads across GPUs (each head is independent), then row-parallel the output projection — same pattern.

Why TP is a within-node story

Each TP all-reduce moves the full hidden-state activation tensor. For a Llama 70B forward pass with hidden=8192, BF16, 4M tokens/step, the per-step volume is gigabytes per layer. NVLink (900 GB/s on H100, 1.8 TB/s on B200) absorbs it. InfiniBand at 400 Gb/s would crush throughput.

Rule of thumb: TP degree ≤ GPUs per NVLink domain. On H100 nodes that's 8. On NVL72 racks it's 72.

EXAMPLE — TP=8 inside a node, why not TP=16

Llama 3 405B: TP=8 within an 8×H100 node (NVLink), PP=16 spanning 16 nodes per replica. Picking TP=16 would require all-reduces over IB — moving multi-GB tensors over 400 Gb/s links each layer. Even with SHARP, the latency tax destroys throughput. NVLink is what makes TP cheap.

REMEMBER

MLP block = column-then-row → one all-reduce per direction.
Attention = head-parallel + row-parallel output proj — same idea.
TP degree ≤ GPUs per NVLink domain. Cross-node TP is malpractice.
Sequence parallel (next chapter) cuts TP's activation memory further.

MODEL PARALLEL · LAYER SPLIT

Pipeline parallel — stages, microbatches, the bubble

TL;DR

Split layers into stages, each on a different GPU. Microbatches flow stage-to-stage like an assembly line. The "pipeline bubble" is the idle time at fill and drain. More microbatches → smaller bubble. Modern schedules (1F1B, interleaved, zero-bubble) attack the bubble from different angles.

The bubble formula

For S stages and M microbatches:

bubble fraction = (S − 1) / (S − 1 + M)

Want bubble < 5%? M ≈ 20·(S−1). With S=16 stages, you need ~300 microbatches per pipeline flush — which constrains your effective batch size and global batch.

Schedule evolution

GPipe: all forwards, then all backwards. Holds activations for the full pipeline depth → activation memory blows up.
1F1B (one-forward-one-backward): alternate forward and backward as soon as a microbatch's backward becomes available. Activation memory bounded by stage depth, not pipeline depth. Megatron default.
Interleaved 1F1B: each stage holds non-contiguous layers (layers 1, 5, 9, 13 on stage 0; 2, 6, 10, 14 on stage 1). More micro-microbatches in flight per macro → smaller bubble.
Zero Bubble Pipeline (Qi 2023): split backward into "input grad" and "weight grad" — reorder so weight grads fill what would otherwise be bubble slots. Near-zero overhead in idealized cases.
DualPipe (DeepSeek V3): bidirectional pipeline — microbatches flow forward AND backward through the stages simultaneously, overlapping compute with the all-to-all comm of MoE. Custom kernels (DeepEP).

Why PP costs little bandwidth

Unlike TP, PP only sends the activation tensor between adjacent stages once per microbatch. That's MBs, not GBs. PP comfortably crosses InfiniBand. PP is your across-node parallelism.

PITFALL — bubble vs batch tension

Cutting the bubble requires more microbatches, but each microbatch increases activation memory at every stage. At a fixed memory budget you face a hard trade: smaller microbatches → more of them → smaller bubble, but more activation memory. Activation checkpointing (chapter 8) is what lets you push microbatch count high without OOM.

REMEMBER

Bubble = (S−1) / (S−1+M). Memorize this.
1F1B is the default schedule. Interleaved 1F1B is what Megatron actually runs.
PP is the across-node parallelism — it sends megabytes, not gigabytes.
Bubble shrinks with more microbatches; activation checkpointing makes that possible.

SEQUENCE PARALLEL · LONG CONTEXT

Sequence & context parallel — when sequences are too long

TL;DR

"Sequence parallel" extends TP by also sharding the sequence dim through LayerNorm/dropout/residual — saves activation memory cheaply. "Context parallel" goes further: shards the sequence inside attention, enabling million-token training. Two competing approaches: ring attention (latency-bound) and DeepSpeed-Ulysses (bandwidth-bound).

Sequence parallel — the cheap activation-memory win

In TP regions, the hidden state is sharded along the feature dim. But LayerNorm, dropout, and residual add are not TP-parallelized — so each rank holds the full activation through them. SP fixes that by also sharding the sequence dim through those ops, with all-gather and reduce-scatter at the SP/TP boundaries. Net effect: cuts activation memory roughly in proportion to TP degree, with negligible throughput cost.

Context parallel — sharding inside attention

True end-to-end sequence sharding, including across the attention computation itself. This is what unlocks multi-million-token training. Two competing approaches you should be able to distinguish:

Ring Attention (Liu 2023)

K/V chunks circulate around a ring of GPUs (point-to-point).
Each GPU computes partial attention against the K/V it currently holds; accumulates via online softmax.
Latency-bound — P2P send/recv each step.
Best for very long contexts (≫ heads count).
Combines with FlashAttention block-wise softmax.

DeepSpeed-Ulysses

Two all-to-all collectives swap parallelism dim between sequence (in attention) and head (in MLP).
Bandwidth-bound — every token crosses the all-to-all twice.
Better for shorter contexts with many heads.
Capped by number of attention heads.

USP (Unified Sequence Parallel) combines them: partition the GPU group into a 2D grid, run Ulysses across one axis and ring across the other. State of the art for very long context training in 2025.

Striped Attention: load-balanced ring — causal mask creates triangular work, so naive ring has stragglers; striped reorders chunks so each GPU computes about the same amount.

REMEMBER

Sequence parallel = cheap activation-memory win on top of TP.
Context parallel = ring (latency-bound) vs Ulysses (bandwidth-bound) vs USP (both).
Ring attention + FlashAttention is what enables million-token training.
If you're training at >128k tokens, you're using one of these.

MOE · EXPERT PARALLEL

Expert parallelism for MoE — and the all-to-all bottleneck

TL;DR

Place experts on different GPUs. Each token routes to top-k experts → two all-to-alls per layer (dispatch + combine). At 256 experts on 256 GPUs the all-to-all dominates compute. Mitigations: capacity factor, topology-aware routing, comm/compute overlap (DualPipe).

Why MoE adds a fifth parallelism dim

An MoE layer replaces one dense FFN with N experts (typically 8 to 256+) and a router that picks the top-k for each token. Different experts hold different weights — so different GPUs hold different experts. Tokens must be shipped to the GPU holding their assigned expert, then the result shipped back. That's an all-to-all dispatch and an all-to-all combine, per MoE layer.

Mitigations

Capacity factor: cap how many tokens any single expert can take. Overflow tokens are dropped (or routed to a fallback). Without this, a popular expert serializes the layer.
Topology-aware routing: prefer experts on GPUs in the same node / NVLink domain. Reduces IB pressure.
DeepSeek V3 DualPipe: bidirectional pipeline that overlaps the all-to-all with compute on the opposite-direction microbatches. Custom kernels (DeepEP) hide ~all the comm.
Expert-level shared weights (DeepSeek MoE): always-active "shared" experts handle generic patterns; routed experts specialize.

EXAMPLE — concrete walkthrough for DeepSeek V3

671B total params, 37B activated per token. 256 routed experts + 1 shared. EP degree = 64 (across nodes). DualPipe + DeepEP overlaps the all-to-all so well that comm overhead is ~zero on H800-class hardware. This is the canonical 2025 MoE-at-scale recipe — reference it explicitly in interviews.

PITFALL — MoE without comm overlap

A naive MoE layer spends 30–60% of step time blocked on all-to-all. If you propose MoE in an interview and don't mention comm/compute overlap (DualPipe-style) or capacity factor, expect pushback. The canonical answer in 2026 is "we'd use DualPipe + DeepEP, like DeepSeek V3".

REMEMBER

MoE adds expert parallelism — a fifth dim on top of DP/TP/PP/CP.
Two all-to-alls per MoE layer: dispatch + combine.
Capacity factor + topology routing + comm overlap make MoE viable.
DeepSeek V3's DualPipe is the public 2025 reference recipe.

PRECISION · NUMERICS

Mixed precision — FP16 / BF16 / FP8 (the H100 era)

TL;DR

BF16 is the modern default — same dynamic range as FP32, no loss scaling needed. FP16 needs loss scaling. FP8 (E4M3 forward, E5M2 backward) doubles throughput on H100/Blackwell but requires fine-grained scaling and FP32 master weights. FP4 is experimental on Blackwell.

The format zoo

Format	Range	Notes
FP16	[6e-5, 65504]	Easy overflow on large activations/grads. Loss scaling needed.
BF16	same exponent as FP32, 7 mantissa bits	Wider range; no loss scaling needed. Default for LLMs.
FP8 E4M3	4-bit exp, 3-bit mantissa	Forward + weights. H100/Blackwell.
FP8 E5M2	5-bit exp, 2-bit mantissa	Backward (needs wider range). Per-tensor scales.
FP4 (Blackwell)	experimental	Microscaling formats (MX).

FP8 — the H100 unlock

FP8 doubles tensor-core throughput vs BF16 and halves activation memory. The cost is operational complexity: per-tensor or per-tile scaling factors must be tracked, FP32 master weights kept, partial sums accumulated in higher precision.

The canonical 2025 recipe is DeepSeek V3's: per-tile scaling (1×128 for activations, 128×128 for weights), online scale computation, FP32 partial-sum promotion every 128 elements, BF16 fallback for embeddings/output head/normalization. Reference this in interviews — it's the public state of the art.

PITFALL — FP8 silent quality regressions

FP8 with naive per-tensor scales loses ~1% on downstream evals. Per-tile scaling closes most of the gap but still leaves a small loss curve gap. Always validate FP8 runs against a BF16 reference at small scale before committing to a long FP8 run. Loss-curve eyeballing is not enough — run downstream evals.

REMEMBER

BF16 is the default. FP16 only if you're stuck on Volta/older.
FP8 = E4M3 forward, E5M2 backward. Per-tile scaling is mandatory at scale.
Critical layers (embedding, output, LN) stay BF16.
FP8 doubles tensor-core throughput — the dominant 2025 H100 efficiency lever.

MEMORY TRADEOFFS

Activation checkpointing & gradient accumulation

TL;DR

Activation checkpointing trades FLOPs for activation memory — recompute during backward instead of storing. Selective recomputation (Megatron) keeps expensive ops, recomputes cheap ones — ~33% extra FLOPs for ~10× memory savings. Gradient accumulation lets you grow effective batch beyond what fits per step.

Activation checkpointing

Save a subset of activations during forward; recompute the rest during backward. Two flavors:

Full block checkpointing: save only block boundaries, recompute everything inside. Simple. Costs ~33% extra FLOPs.
Selective recomputation (Korthikanti 2022): keep expensive ops (attention output, matmul outputs), recompute cheap ops (LayerNorm, GeLU, dropout). Megatron's standard. Roughly 10× activation memory reduction at < 5% throughput cost.

Gradient accumulation

Effective batch = micro-batch × accumulation steps × DP. Run multiple micro-batches sequentially per optimizer step, summing grads, before the all-reduce. Lets you fit when memory caps per-step batch but you still want a large effective batch (good for stable Adam moments).

EXAMPLE — concrete walkthrough at 70B / 8 GPUs

You want effective batch = 4M tokens. Per-GPU memory limits you to 2k tokens/microbatch. With 8 GPUs DP and a single accumulation step, that's 16k tokens/step — not 4M.
Solution: 256 accumulation steps per optimizer update. 2k × 8 × 256 = 4.1M tokens/step. Optimizer step runs 256× less frequently, comm dominated by activation patterns within each microbatch's forward/backward.

REMEMBER

Selective recomputation is the modern default — keep attention outputs, recompute LN/GeLU.
Full-block checkpointing is the fallback if memory still doesn't fit.
Gradient accumulation is how you hit a "4M-token batch" with limited per-step memory.
These two tricks are what let pipeline parallel push microbatch count high enough to shrink the bubble.

COLLECTIVES · COMM PRIMITIVES

The collectives toolkit — AllReduce, AllGather, AllToAll

TL;DR

Every distributed-training scheme is a recipe of collective ops. AllReduce dominates DP. AllGather and ReduceScatter are how ZeRO-3 works. AllToAll is the MoE bottleneck. Cost in ring all-reduce is 2(N−1)/N · M per rank — close to 2M for large N.

Primitive	Result	Cost (ring, M bytes, N ranks)
AllReduce	every rank ends with sum (or other reduction)	2(N−1)/N · M per rank
AllGather	every rank ends with concatenation	(N−1)/N · M
ReduceScatter	reduce + each rank gets a chunk	(N−1)/N · M
Broadcast	one rank → all ranks	O(M log N)
AllToAll	every rank sends a chunk to every other	(N−1)/N · M. Critical for MoE.

The identity to memorize: AllReduce = ReduceScatter + AllGather. This is why ZeRO-2 is "free" relative to DDP — replacing one all-reduce with a reduce-scatter (then later an all-gather where the data was needed anyway) keeps total volume identical.

REMEMBER

Ring AllReduce: ~2M per rank, regardless of N. Bandwidth-bound.
AllReduce = ReduceScatter + AllGather (the fundamental identity).
AllToAll is the MoE / Ulysses primitive — N² messages but linear total volume.
Tree algorithms beat ring for very small messages (latency-bound regime).

HARDWARE · INTERCONNECT

Hardware reality — NVLink, NVSwitch, IB, NCCL, SHARP

TL;DR

NVLink (within node) and InfiniBand (across node) form a two-tier bandwidth hierarchy. NVSwitch makes 8-GPU nodes look like one big GPU; NVL72 extends that to 72 GPUs in a rack. NCCL is the collectives library that abstracts it all. SHARP does in-network reduction on IB switches.

NCCL: NVIDIA's collectives library. GPU-direct via NVLink/PCIe/IB. The substrate everything sits on.
NVLink: GPU-to-GPU direct. H100: 900 GB/s aggregate per GPU. B200: 1.8 TB/s.
NVSwitch: full-mesh of NVLinks within a node (8 GPUs typical). Every GPU talks to every other at full NVLink bandwidth.
NVL72 (Blackwell): NVLink Switch extends NVSwitch fabric to 72 GPUs in a single rack. Lets you do TP=72 (vs 8 on H100) or fit much larger TP×PP combos within one rack.
InfiniBand: across-node interconnect. ConnectX-7 NDR = 400 Gb/s. Topology is fat-tree, often "rail-optimized" (each GPU has its own NIC).
SHARP: in-network reduction on IB switches. The switch sums fragments and broadcasts the result, eliminating one half of all-reduce traffic. Big win for small messages, less so for huge ones.

The bandwidth cliff

The reason TP stays within node is the bandwidth cliff: NVLink is ~20× the bandwidth of a single IB link. Every parallelism choice is a placement problem on this two-tier hierarchy.

REMEMBER

NVLink within node, IB across node. Two tiers.
NVSwitch = full mesh inside a node. NVL72 = full mesh inside a rack.
SHARP halves all-reduce traffic — useful for small messages.
Rail-optimized fat-tree is the standard cluster topology in 2025.

DURABILITY · CHECKPOINTING

Checkpointing strategies — sync, async, sharded

TL;DR

Sync checkpointing pauses training for 10–30 minutes on a 70B model — unacceptable at scale. Async (NVMe + background upload) cuts pause to under a minute. Sharded (each rank writes its own slice) is mandatory for trillion-param models. Frequency is set by MTBF — every 30 min to a few hours.

Sync: all ranks pause, write to shared FS. 70B model write ~10–30 min on slow FS. Only acceptable for small jobs.
Async: dump to local NVMe (< 1 min for 70B), separate process uploads to S3/GCS. Training continues immediately.
Sharded: each rank writes its own shard. Resume requires same parallelism config — or a re-shard tool. Megatron's dist-ckpt and TorchSnapshot handle resharding.
In-memory replication: state copied to other nodes for fast recovery without disk. Emerging at frontier labs.

Frequency tuning — MTBF dictates cadence

At 10k+ GPUs, hardware fails every few hours. Checkpoint cadence should be set so expected lost work per failure is acceptable. Llama 3 405B writeup: checkpoint every ~30 min → ~hour, which dominates the engineering pain budget.

EXAMPLE — async checkpoint pipeline

Step N completes. Sharded NVMe dump fires (each rank writes ~17 GB / 70B/8). Training continues with no observable pause. Background process tars and uploads to S3 over the next few minutes. Step N+200 complete before the upload finishes — that's fine. Recovery reads from NVMe if the node survived; from S3 otherwise.

REMEMBER

Async + sharded + NVMe-tier-1 + S3-tier-2 is the modern recipe.
Sync to shared FS is fine for <1k-GPU runs, malpractice at 10k+.
Checkpoint every ~30 min — set by MTBF, not by intuition.
Resharding tools matter when you change parallelism config mid-run.

SCALE · OPERATIONS

Failures at 10k+ GPU scale — what kills naive runs

TL;DR

At 10k+ GPUs, the failure rate is once every few hours. NCCL hangs, silent data corruption, ECC errors, network stragglers. A 70B run on 100 GPUs is "build it and run it"; a 405B run on 16k GPUs is "build the failure-handling system that happens to also train a model". This chapter is the public failure-handling toolkit from Llama 3 + xAI Colossus writeups.

The failure modes

NCCL hang: one rank dies mid-collective; the rest spin forever. Without a watchdog, the whole job stalls indefinitely.
Silent data corruption (SDC): H100 SDC rates are non-trivial at fleet scale. A rank computes the wrong result without any error signal.
Stragglers: ring all-reduce is bottlenecked by the slowest rank. A degraded NIC or thermally throttled GPU drags the whole job.
OOM cascades: a single OOM on rank N takes down the whole job, requires restart from checkpoint, possibly with different parallelism config.
Loss spikes: a bad batch, a numerical instability, or rank corruption causes loss to diverge.

The mitigation toolkit

NCCL watchdog: monitors collective progress; aborts hung ranks with timeout. Without this, debugging is impossible.
Heartbeat-based stale rank detection: per-rank heartbeats on a sidecar; missing beat → abort + replace.
SDC detection: gradient-norm spike checks, hash-based comparison across DP replicas, periodic loss-on-validation-batch sanity checks. A sudden 10× gradient norm with normal-looking loss is a strong SDC signal.
ECC + scrubbing: in-band ECC on HBM auto-corrects single-bit errors; out-of-band scrubbing reports double-bit errors so you can preemptively replace the GPU.
Graceful node replacement: hot spares standing by; on failure, restart the affected stage from last checkpoint while other stages continue.
Async checkpoint to NVMe + lazy upload: NVMe write is < 1 min; S3/GCS upload happens in background.
Last-known-good restart: if loss diverges, restart from the most recent healthy checkpoint, not just the most recent.
Network straggler detection: per-link latency monitoring; rebalance job placement when degraded links found.

PITFALL — autoscaling lag and silent data corruption

Two classes of failure are easy to miss in interview answers:
(1) Autoscaling lag: GPU spin-up takes minutes. If your hot-spare strategy assumes instant replacement, you'll lose hours per failure. Pre-warm spares and keep them in the same NCCL group.
(2) Silent data corruption: an H100 fleet at 10k+ scale will have undetected bit-flips. Loss curves look fine. You only notice via hash-comparison across DP replicas or periodic validation-batch sanity. Naming SDC unprompted is a strong signal in OpenAI/Anthropic loops.

REMEMBER

10k+ GPU runs fail every few hours. Plan for it.
NCCL watchdog + heartbeats + hot spares are non-negotiable.
SDC is real; detect via cross-DP hash compare and grad-norm spikes.
"Last-known-good restart" beats "most-recent restart" when loss diverges.

0 → hero reading path for distributed training

foundation HF docs — Multi-GPU training
foundation DeepSpeed tutorials — ZeRO 1/2/3 explained
foundation PyTorch FSDP docs
build nanoGPT with DDP — modify to FSDP — modify to TP
build Run a 7B fine-tune on 8 GPUs with FSDP; profile with PyTorch profiler
depth ZeRO paper (Rajbhandari 2019)
depth Megatron-LM (Shoeybi 2019)
depth Sequence parallelism (Korthikanti 2022)
depth Ring Attention (Liu 2023)
depth DeepSpeed-Ulysses
depth DeepSeek V3 technical report — DualPipe, FP8 recipe
depth Llama 3 paper — practical 16k-GPU training
depth Simon Boehm — Data parallelism overview
depth Horace He — Thonking.ai on PyTorch internals

Distributed training quiz — readiness check

You have a 70B model on 8 H100s. What's your parallelism plan?
Show answer
Single node, NVLink. ZeRO-3 / FSDP across 8 GPUs. Per-GPU memory: bf16 weights ~17.5 GB + grads ~17.5 GB + optimizer state ~17.5 GB (sharded) + activations + KV. Add activation checkpointing for moderate batch.
Now 405B on 16384 H100s — what's the plan?
Show answer
3D: TP=8 (within node), PP=16 (across 16 nodes per replica), DP=128 (16384/8/16). Microbatches per pipeline = enough to keep bubble < 5%. Selective activation recomputation. ZeRO-1 on top for optimizer-state sharding.
DDP vs FSDP — when each?
Show answer
DDP if model + optimizer state fit on one GPU (small models, < ~10B). FSDP otherwise. FSDP costs ~1.5× DDP comm.
Why does TP need high bandwidth?
Show answer
Two all-reduces per layer over the full hidden state. For Llama 70B (hidden=8192, bf16, batch=4M tokens/step), each TP all-reduce moves GBs/s. NVLink (900 GB/s) handles this; cross-node IB would crush it.
What's the pipeline bubble formula?
Show answer
Bubble fraction = (p − 1) / (p − 1 + m), where p = stages, m = microbatches. Increase m to reduce. Interleaved 1F1B partitions layers non-contiguously to shrink the bubble further.
Explain MoE all-to-all bottleneck.
Show answer
Each token routed to top-k experts on different GPUs. Two all-to-alls per layer (dispatch + combine). At 256 experts on 256 GPUs, all-to-all dominates compute. Mitigations: capacity factor, topology-aware routing, comm/compute overlap (DualPipe).
Bandwidth requirements for Llama 405B training?
Show answer
TP all-reduces: NVLink (900 GB/s) handles within-node. PP cross-node: only activation tensor between adjacent stages (~MB), low BW. DP gradient all-reduce: 405B params bf16 = 810 GB; over 16k GPUs in fat-tree IB = seconds per step.
What goes wrong at 10k+ GPU scale that doesn't at 100?
Show answer
Failures every few hours (NIC, GPU ECC, OOM). Need fast checkpointing + restart, watchdog on NCCL hangs, async retries, hot-swap of failed nodes. Loss-spike detection that pauses training. Network straggler detection. Silent data corruption (SDC) at H100 fleet scale.
What does sequence parallel shard?
Show answer
In TP regions, also shard the sequence dim during LN/dropout/residual (where TP doesn't help). All-gather + reduce-scatter on TP/SP boundaries. Cuts activation memory; doesn't change throughput much.
Ring attention vs DeepSpeed-Ulysses — when each?
Show answer
Ring: P2P circulation of K/V chunks; latency-bound; great for very long contexts. Ulysses: 2 all-to-alls swap parallelism dim (sequence ↔ head); bandwidth-bound; better for short contexts with many heads. USP combines both via 2D grid.
16 bytes/param breakdown?
Show answer
fp32 master weights (4) + fp32 Adam m (4) + fp32 Adam v (4) + bf16 weights (2) + bf16 grads (2) = 16. ZeRO sharding distributes these across DP ranks.
Why is FP8 training tricky?
Show answer
FP8 has only 8 bits — limited dynamic range. Two formats (E4M3 fwd, E5M2 bwd) for different ranges. Need careful per-tensor or per-tile scaling, FP32 master weights, FP32 partial-sum accumulation. DeepSeek V3 fine-grained per-tile scaling is the canonical recipe.
NCCL hang — diagnose and fix.
Show answer
Symptom: training stalls; one rank not making progress on a collective. Causes: dead rank (GPU ECC, OOM), network partition, deadlock from mismatched collective on different ranks. Fix: NCCL watchdog (NCCL_TIMEOUT) aborts the job; restart from checkpoint, replace bad node.
Why interleaved 1F1B instead of vanilla 1F1B?
Show answer
Each stage holds non-contiguous layers (e.g., layers 1, 5, 9 on stage 0; 2, 6, 10 on stage 1). More micro-microbatches in flight per macro-microbatch → smaller pipeline bubble. Megatron uses this.
Difference between gradient accumulation and increasing batch size?
Show answer
Effective batch = micro-batch × accumulation × DP. Same effective batch, but accumulation lets you fit when memory limits per-step batch. Can be combined with mixed precision and activation checkpointing for further memory reduction.
What's selective activation recomputation?
Show answer
Recompute only cheap ops (LN, GELU, dropout) during backward; keep expensive (attention, matmul outputs) in memory. Megatron's standard. ~33% extra FLOPs vs full recompute, but 10× memory savings on activations.
Difference between ZeRO-Infinity, ZeRO-Offload, and ZeRO-3?
Show answer
ZeRO-3: shard params/grads/optimizer across DP ranks (all in HBM). ZeRO-Offload: offload optimizer state + parts of gradient to CPU RAM. ZeRO-Infinity: extends to NVMe. Used when model truly doesn't fit in HBM; slow due to CPU/NVMe bandwidth.
NVL72 vs PCIe — what changes?
Show answer
NVLink Switch (NVL72) gives 72 GPUs per "node" with full NVLink bandwidth. Lets you do TP=72 (vs 8 on H100), or much larger TP × PP combos within one rack. Fewer cross-IB hops needed for many parallelism plans.
What is SHARP and why does it matter?
Show answer
NVIDIA's in-network reduction on InfiniBand switches. The switch performs the reduction (sum) and broadcasts the result, eliminating one half of all-reduce comm time. Speeds up small all-reduces; less impact on huge ones.
Async checkpointing — how does it work?
Show answer
Dump model state to local NVMe (fast — < 1 min for 70B). Separate process uploads to durable storage (S3/GCS) in background. Training continues. Reduces pause from ~30 min (sync to slow FS) to ~1 min.