Distributed training

DDP — the floor everyone starts from

Replicate the model on each GPU, shard the batch. After backward, all-reduce gradients across replicas. Memory-heavy: each GPU holds the full model + grads + optimizer state. Comm cost = one all-reduce per step on the gradient tensor.

DDP is the right answer when the entire training state fits on one GPU with room for activations. Below ~7B parameters in BF16, fine. Above that, you're allocating optimizer state you can't afford.

The 16 bytes/param breakdown

For Adam mixed-precision (the dominant LLM recipe), the per-parameter memory accounting from the ZeRO paper (Table 1):

The three ZeRO stages

From Rajbhandari et al. 2019 (arxiv 1910.02054). ZeRO progressively partitions training state across DP ranks instead of replicating.

The 1.5× number is from ZeRO paper Table 5 — frequently misquoted as "2×". Forward needs an all-gather on each layer's params (used, then freed). Backward needs another all-gather (re-fetch what was freed) plus a reduce-scatter on grads. Three collective passes vs DDP's one all-reduce, but the reduce-scatter and all-reduce cost the same per byte.

FSDP — PyTorch's production ZeRO-3

FSDP wraps modules; manages all-gather/reduce-scatter automatically per-wrap-unit. FSDP-2 (PyTorch 2.x) introduces per-parameter sharding (rather than per-flat-buffer), which composes cleanly with TP and is more memory-efficient for irregular shapes.

ZeRO-Offload and ZeRO-Infinity

When even ZeRO-3 doesn't fit, push state out of HBM:

• ZeRO-Offload: optimizer state and parts of gradient live in CPU RAM. Optimizer step runs on CPU. Slow but unblocks training a 13B model on a single 24 GB GPU.
• ZeRO-Infinity: extends to NVMe. Used as an absolute last resort; CPU/NVMe bandwidth caps throughput hard.

The column-then-row trick

From Shoeybi et al. 2019 (arxiv 1909.08053). For a two-matmul block Z = GeLU(X · A) · B, choose the shard axes so the GeLU is local:

• Column-parallel A: each GPU holds A_i (a column slice). Computes Y_i = GeLU(X · A_i). No comm — GeLU is elementwise.
• Row-parallel B: each GPU holds B_i (a row slice). Computes partial Z_i. All-reduce (sum) to combine.

Net cost: one all-reduce in forward, one in backward, per MLP block. For attention, shard heads across GPUs (each head is independent), then row-parallel the output projection — same pattern.

Why TP is a within-node story

Each TP all-reduce moves the full hidden-state activation tensor. For a Llama 70B forward pass with hidden=8192, BF16, 4M tokens/step, the per-step volume is gigabytes per layer. NVLink (900 GB/s on H100, 1.8 TB/s on B200) absorbs it. InfiniBand at 400 Gb/s would crush throughput.

Rule of thumb: TP degree ≤ GPUs per NVLink domain. On H100 nodes that's 8. On NVL72 racks it's 72.

The bubble formula

For S stages and M microbatches:

bubble fraction = (S − 1) / (S − 1 + M)

Want bubble < 5%? M ≈ 20·(S−1). With S=16 stages, you need ~300 microbatches per pipeline flush — which constrains your effective batch size and global batch.

Schedule evolution

• GPipe: all forwards, then all backwards. Holds activations for the full pipeline depth → activation memory blows up.
• 1F1B (one-forward-one-backward): alternate forward and backward as soon as a microbatch's backward becomes available. Activation memory bounded by stage depth, not pipeline depth. Megatron default.
• Interleaved 1F1B: each stage holds non-contiguous layers (layers 1, 5, 9, 13 on stage 0; 2, 6, 10, 14 on stage 1). More micro-microbatches in flight per macro → smaller bubble.
• Zero Bubble Pipeline (Qi 2023): split backward into "input grad" and "weight grad" — reorder so weight grads fill what would otherwise be bubble slots. Near-zero overhead in idealized cases.
• DualPipe (DeepSeek V3): bidirectional pipeline — microbatches flow forward AND backward through the stages simultaneously, overlapping compute with the all-to-all comm of MoE. Custom kernels (DeepEP).

Why PP costs little bandwidth

Unlike TP, PP only sends the activation tensor between adjacent stages once per microbatch. That's MBs, not GBs. PP comfortably crosses InfiniBand. PP is your across-node parallelism.

Sequence parallel — the cheap activation-memory win

In TP regions, the hidden state is sharded along the feature dim. But LayerNorm, dropout, and residual add are not TP-parallelized — so each rank holds the full activation through them. SP fixes that by also sharding the sequence dim through those ops, with all-gather and reduce-scatter at the SP/TP boundaries. Net effect: cuts activation memory roughly in proportion to TP degree, with negligible throughput cost.

Context parallel — sharding inside attention

True end-to-end sequence sharding, including across the attention computation itself. This is what unlocks multi-million-token training. Two competing approaches you should be able to distinguish:

USP (Unified Sequence Parallel) combines them: partition the GPU group into a 2D grid, run Ulysses across one axis and ring across the other. State of the art for very long context training in 2025.

Striped Attention: load-balanced ring — causal mask creates triangular work, so naive ring has stragglers; striped reorders chunks so each GPU computes about the same amount.

Why MoE adds a fifth parallelism dim

An MoE layer replaces one dense FFN with N experts (typically 8 to 256+) and a router that picks the top-k for each token. Different experts hold different weights — so different GPUs hold different experts. Tokens must be shipped to the GPU holding their assigned expert, then the result shipped back. That's an all-to-all dispatch and an all-to-all combine, per MoE layer.

Mitigations

• Capacity factor: cap how many tokens any single expert can take. Overflow tokens are dropped (or routed to a fallback). Without this, a popular expert serializes the layer.
• Topology-aware routing: prefer experts on GPUs in the same node / NVLink domain. Reduces IB pressure.
• DeepSeek V3 DualPipe: bidirectional pipeline that overlaps the all-to-all with compute on the opposite-direction microbatches. Custom kernels (DeepEP) hide ~all the comm.
• Expert-level shared weights (DeepSeek MoE): always-active "shared" experts handle generic patterns; routed experts specialize.

The format zoo

FP8 — the H100 unlock

FP8 doubles tensor-core throughput vs BF16 and halves activation memory. The cost is operational complexity: per-tensor or per-tile scaling factors must be tracked, FP32 master weights kept, partial sums accumulated in higher precision.

The canonical 2025 recipe is DeepSeek V3's: per-tile scaling (1×128 for activations, 128×128 for weights), online scale computation, FP32 partial-sum promotion every 128 elements, BF16 fallback for embeddings/output head/normalization. Reference this in interviews — it's the public state of the art.

Activation checkpointing

Save a subset of activations during forward; recompute the rest during backward. Two flavors:

• Full block checkpointing: save only block boundaries, recompute everything inside. Simple. Costs ~33% extra FLOPs.
• Selective recomputation (Korthikanti 2022): keep expensive ops (attention output, matmul outputs), recompute cheap ops (LayerNorm, GeLU, dropout). Megatron's standard. Roughly 10× activation memory reduction at < 5% throughput cost.

Gradient accumulation

Effective batch = micro-batch × accumulation steps × DP. Run multiple micro-batches sequentially per optimizer step, summing grads, before the all-reduce. Lets you fit when memory caps per-step batch but you still want a large effective batch (good for stable Adam moments).

The identity to memorize: AllReduce = ReduceScatter + AllGather. This is why ZeRO-2 is "free" relative to DDP — replacing one all-reduce with a reduce-scatter (then later an all-gather where the data was needed anyway) keeps total volume identical.

• NCCL: NVIDIA's collectives library. GPU-direct via NVLink/PCIe/IB. The substrate everything sits on.
• NVLink: GPU-to-GPU direct. H100: 900 GB/s aggregate per GPU. B200: 1.8 TB/s.
• NVSwitch: full-mesh of NVLinks within a node (8 GPUs typical). Every GPU talks to every other at full NVLink bandwidth.
• NVL72 (Blackwell): NVLink Switch extends NVSwitch fabric to 72 GPUs in a single rack. Lets you do TP=72 (vs 8 on H100) or fit much larger TP×PP combos within one rack.
• InfiniBand: across-node interconnect. ConnectX-7 NDR = 400 Gb/s. Topology is fat-tree, often "rail-optimized" (each GPU has its own NIC).
• SHARP: in-network reduction on IB switches. The switch sums fragments and broadcasts the result, eliminating one half of all-reduce traffic. Big win for small messages, less so for huge ones.

The bandwidth cliff

The reason TP stays within node is the bandwidth cliff: NVLink is ~20× the bandwidth of a single IB link. Every parallelism choice is a placement problem on this two-tier hierarchy.

• Sync: all ranks pause, write to shared FS. 70B model write ~10–30 min on slow FS. Only acceptable for small jobs.
• Async: dump to local NVMe (< 1 min for 70B), separate process uploads to S3/GCS. Training continues immediately.
• Sharded: each rank writes its own shard. Resume requires same parallelism config — or a re-shard tool. Megatron's dist-ckpt and TorchSnapshot handle resharding.
• In-memory replication: state copied to other nodes for fast recovery without disk. Emerging at frontier labs.

Frequency tuning — MTBF dictates cadence

At 10k+ GPUs, hardware fails every few hours. Checkpoint cadence should be set so expected lost work per failure is acceptable. Llama 3 405B writeup: checkpoint every ~30 min → ~hour, which dominates the engineering pain budget.

The failure modes

• NCCL hang: one rank dies mid-collective; the rest spin forever. Without a watchdog, the whole job stalls indefinitely.
• Silent data corruption (SDC): H100 SDC rates are non-trivial at fleet scale. A rank computes the wrong result without any error signal.
• Stragglers: ring all-reduce is bottlenecked by the slowest rank. A degraded NIC or thermally throttled GPU drags the whole job.
• OOM cascades: a single OOM on rank N takes down the whole job, requires restart from checkpoint, possibly with different parallelism config.
• Loss spikes: a bad batch, a numerical instability, or rank corruption causes loss to diverge.

The mitigation toolkit

1. NCCL watchdog: monitors collective progress; aborts hung ranks with timeout. Without this, debugging is impossible.
2. Heartbeat-based stale rank detection: per-rank heartbeats on a sidecar; missing beat → abort + replace.
3. SDC detection: gradient-norm spike checks, hash-based comparison across DP replicas, periodic loss-on-validation-batch sanity checks. A sudden 10× gradient norm with normal-looking loss is a strong SDC signal.
4. ECC + scrubbing: in-band ECC on HBM auto-corrects single-bit errors; out-of-band scrubbing reports double-bit errors so you can preemptively replace the GPU.
5. Graceful node replacement: hot spares standing by; on failure, restart the affected stage from last checkpoint while other stages continue.
6. Async checkpoint to NVMe + lazy upload: NVMe write is < 1 min; S3/GCS upload happens in background.
7. Last-known-good restart: if loss diverges, restart from the most recent healthy checkpoint, not just the most recent.
8. Network straggler detection: per-link latency monitoring; rebalance job placement when degraded links found.

Distributed training quiz — readiness check

1. You have a 70B model on 8 H100s. What's your parallelism plan?
   Show answer

   Single node, NVLink. ZeRO-3 / FSDP across 8 GPUs. Per-GPU memory: bf16 weights ~17.5 GB + grads ~17.5 GB + optimizer state ~17.5 GB (sharded) + activations + KV. Add activation checkpointing for moderate batch.

2. Now 405B on 16384 H100s — what's the plan?
   Show answer

   3D: TP=8 (within node), PP=16 (across 16 nodes per replica), DP=128 (16384/8/16). Microbatches per pipeline = enough to keep bubble < 5%. Selective activation recomputation. ZeRO-1 on top for optimizer-state sharding.

3. DDP vs FSDP — when each?
   Show answer

   DDP if model + optimizer state fit on one GPU (small models, < ~10B). FSDP otherwise. FSDP costs ~1.5× DDP comm.

4. Why does TP need high bandwidth?
   Show answer

   Two all-reduces per layer over the full hidden state. For Llama 70B (hidden=8192, bf16, batch=4M tokens/step), each TP all-reduce moves GBs/s. NVLink (900 GB/s) handles this; cross-node IB would crush it.

5. What's the pipeline bubble formula?
   Show answer

   Bubble fraction = (p − 1) / (p − 1 + m), where p = stages, m = microbatches. Increase m to reduce. Interleaved 1F1B partitions layers non-contiguously to shrink the bubble further.

6. Explain MoE all-to-all bottleneck.
   Show answer

   Each token routed to top-k experts on different GPUs. Two all-to-alls per layer (dispatch + combine). At 256 experts on 256 GPUs, all-to-all dominates compute. Mitigations: capacity factor, topology-aware routing, comm/compute overlap (DualPipe).

7. Bandwidth requirements for Llama 405B training?
   Show answer

   TP all-reduces: NVLink (900 GB/s) handles within-node. PP cross-node: only activation tensor between adjacent stages (~MB), low BW. DP gradient all-reduce: 405B params bf16 = 810 GB; over 16k GPUs in fat-tree IB = seconds per step.

8. What goes wrong at 10k+ GPU scale that doesn't at 100?
   Show answer

   Failures every few hours (NIC, GPU ECC, OOM). Need fast checkpointing + restart, watchdog on NCCL hangs, async retries, hot-swap of failed nodes. Loss-spike detection that pauses training. Network straggler detection. Silent data corruption (SDC) at H100 fleet scale.

9. What does sequence parallel shard?
   Show answer

   In TP regions, also shard the sequence dim during LN/dropout/residual (where TP doesn't help). All-gather + reduce-scatter on TP/SP boundaries. Cuts activation memory; doesn't change throughput much.

10. Ring attention vs DeepSpeed-Ulysses — when each?
    Show answer

    Ring: P2P circulation of K/V chunks; latency-bound; great for very long contexts. Ulysses: 2 all-to-alls swap parallelism dim (sequence ↔ head); bandwidth-bound; better for short contexts with many heads. USP combines both via 2D grid.

11. 16 bytes/param breakdown?
    Show answer

    fp32 master weights (4) + fp32 Adam m (4) + fp32 Adam v (4) + bf16 weights (2) + bf16 grads (2) = 16. ZeRO sharding distributes these across DP ranks.

12. Why is FP8 training tricky?
    Show answer

    FP8 has only 8 bits — limited dynamic range. Two formats (E4M3 fwd, E5M2 bwd) for different ranges. Need careful per-tensor or per-tile scaling, FP32 master weights, FP32 partial-sum accumulation. DeepSeek V3 fine-grained per-tile scaling is the canonical recipe.

13. NCCL hang — diagnose and fix.
    Show answer

    Symptom: training stalls; one rank not making progress on a collective. Causes: dead rank (GPU ECC, OOM), network partition, deadlock from mismatched collective on different ranks. Fix: NCCL watchdog (NCCL_TIMEOUT) aborts the job; restart from checkpoint, replace bad node.

14. Why interleaved 1F1B instead of vanilla 1F1B?
    Show answer

    Each stage holds non-contiguous layers (e.g., layers 1, 5, 9 on stage 0; 2, 6, 10 on stage 1). More micro-microbatches in flight per macro-microbatch → smaller pipeline bubble. Megatron uses this.

15. Difference between gradient accumulation and increasing batch size?
    Show answer

    Effective batch = micro-batch × accumulation × DP. Same effective batch, but accumulation lets you fit when memory limits per-step batch. Can be combined with mixed precision and activation checkpointing for further memory reduction.

16. What's selective activation recomputation?
    Show answer

    Recompute only cheap ops (LN, GELU, dropout) during backward; keep expensive (attention, matmul outputs) in memory. Megatron's standard. ~33% extra FLOPs vs full recompute, but 10× memory savings on activations.

17. Difference between ZeRO-Infinity, ZeRO-Offload, and ZeRO-3?
    Show answer

    ZeRO-3: shard params/grads/optimizer across DP ranks (all in HBM). ZeRO-Offload: offload optimizer state + parts of gradient to CPU RAM. ZeRO-Infinity: extends to NVMe. Used when model truly doesn't fit in HBM; slow due to CPU/NVMe bandwidth.

18. NVL72 vs PCIe — what changes?
    Show answer

    NVLink Switch (NVL72) gives 72 GPUs per "node" with full NVLink bandwidth. Lets you do TP=72 (vs 8 on H100), or much larger TP × PP combos within one rack. Fewer cross-IB hops needed for many parallelism plans.

19. What is SHARP and why does it matter?
    Show answer

    NVIDIA's in-network reduction on InfiniBand switches. The switch performs the reduction (sum) and broadcasts the result, eliminating one half of all-reduce comm time. Speeds up small all-reduces; less impact on huge ones.

20. Async checkpointing — how does it work?
    Show answer

    Dump model state to local NVMe (fast — < 1 min for 70B). Separate process uploads to durable storage (S3/GCS) in background. Training continues. Reduces pause from ~30 min (sync to slow FS) to ~1 min.