LLM training & RLHF

Chinchilla in one equation, and why nobody obeys it

Hoffmann et al. (2022) fit pretraining loss as a function of parameters $N$ and tokens $D$ given a compute budget $C \approx 6ND$ FLOPs:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}, \quad \alpha \approx 0.34, \; \beta \approx 0.28$$

Setting $\partial L/\partial N = \partial L/\partial D$ under the budget constraint $C = 6ND$ gives the Chinchilla-optimal allocation $D^* \approx 20 N$. The 2024–2026 frontier deliberately violates this on the data side because training compute is a one-time cost, inference compute is forever. Sardana & Frankle (arXiv 2401.00448) made this rigorous as the "Beyond Chinchilla-Optimal" rule: minimize total cost over expected inference volume, not training loss.

Data sources — what goes in

Frontier mixtures pull from six buckets, in roughly decreasing volume:

• Web crawl — CommonCrawl, RefinedWeb, FineWeb v2, FineWeb-Edu, Nemotron-CC v2.
• Code — GitHub, StackExchange, The Stack v3, public Jupyter, Codeforces archives.
• Books, Wikipedia, papers — ArXiv (LaTeX source preferred), S2ORC, PubMed Central.
• Curated forums & math — OpenWebMath v2, FineMath, AoPS forums, Lean/Mathlib.
• Multilingual — mC4, CulturaX, MADLAD-400.
• Synthetic — Phi-style textbooks, Cosmopedia v3, model-generated reasoning chains, persona-tagged QA. In 2026 this typically exceeds 20% of the mix.

Quality filtering — six-stage pipeline

1. URL/domain filtering — block adult content, low-quality TLDs, known content farms.
2. Language ID via fastText or CLD3 — keep target languages only.
3. Heuristic (Gopher) rules — avg word length 3–10, % lines ending in punctuation, % stop words, % alphabetic chars.
4. Repetition removal — drop docs with excessive line/paragraph repetition.
5. Toxicity / PII filters.
6. Quality classifier — small classifier (typically ~1B params) trained on labels distilled from a teacher LLM scoring "educational value"; score every doc and threshold.

Deduplication — three layers stacked

• Exact dedup — hash per document or paragraph (Bloom filter at scale).
• MinHash LSH — compute MinHash signatures over $n$-gram shingles ($n = 5$ typical). With $b$ bands of $r$ hashes each, the collision probability at Jaccard $s$ is $1 - (1 - s^r)^b$ — an S-curve with knee near $s \approx (1/b)^{1/r}$. For $\sim 5\mathrm{T}$ documents, $b = 10$, $r = 9$ is the canonical setup (knee at $s \approx 0.77$). Verify candidates with exact Jaccard.
• SemDeDup (Abbas 2023, arXiv 2303.09540) — cluster embeddings with k-means, dedup within clusters by cosine. Catches paraphrases and translations MinHash misses.
• Suffix array dedup (Lee 2022, arXiv 2107.06499) — exact dedup of long substrings within and across documents. Strips license text, navigation bars, boilerplate.

Mixing weights & curricula

Up-sample high-quality (Wikipedia, books, code), down-sample bulk web. DoReMi (Xie 2023, arXiv 2305.10429) trains a small reference + proxy and uses Group DRO to find weights $\boldsymbol{\alpha}$ minimizing worst-case excess loss:

$$\boldsymbol{\alpha}^* = \arg\min_{\boldsymbol{\alpha}} \max_{i \in \mathcal{D}} \left[ L_i(\theta_{\boldsymbol{\alpha}}) - L_i(\theta_{\text{ref}}) \right]$$

The reference is your best small model; proxy weights chase the worst-performing domain. DataComp-LM (Li 2024, arXiv 2406.11794) sweeps mixtures empirically. Most labs combine: DoReMi for initial weights, then late-stage anneal with a hand-tuned mid-training mix.

Mid-training — the new norm

Mid-training continues causal LM training but with a different mixture — heavy up-weight on reasoning, math, code, possibly synthetic chain-of-thought. Typically combined with a WSD (warmup-stable-decay) schedule where the decay phase coincides with the mixture shift. The model becomes "reasoning-ready" without yet being instruction-tuned. Token budgets: $5\text{–}25\%$ of total pretraining tokens.

Continual pretraining (CPT) — adapting a frozen base

Take a pretrained model and continue on new data — new language, new domain, new time period. The risk is catastrophic forgetting; the standard mitigations are:

• Replay — mix old data, 5–30% of original mixture.
• Lower LR — $1/10$ to $1/100$ of pretraining peak.
• LR schedule — small warmup, then decay matching the new token budget.
• EWC-style regularization — penalize movement on weights with large Fisher information $F_i$: $\mathcal{L}_{\text{EWC}} = \mathcal{L} + \tfrac{\lambda}{2} \sum_i F_i (\theta_i - \theta_i^*)^2$. Rarely used at frontier scale; replay typically suffices.

The mechanics

Single- or multi-turn instruction-response pairs. Loss is cross-entropy on response tokens; the prompt mask zeroes out prompt positions. Formally, for a sequence $y = (y_1, \dots, y_T)$ partitioned into prompt $\mathcal{P}$ and response $\mathcal{R}$:

$$\mathcal{L}_{\text{SFT}} = -\frac{1}{|\mathcal{R}|} \sum_{t \in \mathcal{R}} \log \pi_\theta(y_t \mid y_{

Modern SFT datasets fall into three buckets:

• Synthetic from a stronger model — Self-Instruct, Evol-Instruct, OpenHermes, Tülu mixes, NuminaMath.
• Human-written — OpenAssistant, ShareGPT-style.
• Filtered/curated mixtures from existing public corpora; rejection-sampled completions from a strong base.

LIMA hypothesis vs Tülu 3/4

LIMA (arXiv 2305.11206) — "less is more for alignment" — diversity $>$ scale beyond ~10k–100k examples. Capabilities live in the base model; SFT only teaches format/instruction-following.

Tülu 3 (arXiv 2411.15124) and Tülu 4 (arXiv 2511.18402) updated this: up to 1M+ examples helps when domains are diverse (math, code, IF, safety, multilingual, tool-use). The 2026 statement is "diversity matters more than count, but more diversity is more count."

Few-shot vs zero-shot tradeoff

Pretraining gives ICL (in-context learning) ability for free. SFT collapses the model toward instruction-following at zero-shot, which can sometimes hurt few-shot performance. The fix is a small fraction of "raw completion" examples in the SFT mix.

Original deep RLHF — Christiano 2017

"Deep RL from Human Preferences" (Christiano et al. 2017, arXiv 1706.03741) — predates InstructGPT by 5 years. Demonstrated RLHF on Atari and MuJoCo with a Bradley-Terry preference model and PPO. If an interviewer asks "who invented this?" — name Christiano.

RLHF / PPO — the LLM-era recipe

(Ouyang 2022, arXiv 2203.02155 — InstructGPT.) Three stages:

1. Reward model on $(x, y_w, y_l)$ pairs with Bradley-Terry: $$P(y_w \succ y_l \mid x) = \sigma\big( r_\phi(x, y_w) - r_\phi(x, y_l) \big), \quad \mathcal{L}_{\text{RM}} = -\mathbb{E}\big[\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))\big].$$
2. RL fine-tune policy $\pi_\theta$ with PPO against $r_\phi$ plus a per-token KL penalty to the SFT reference: $$r_{\text{total}}(x, y) = r_\phi(x, y) - \beta \cdot \mathrm{KL}\big(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\big).$$
3. PPO clipped objective with importance ratio $\rho_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$ and advantage $\hat A_t$ (GAE): $$\mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t \Big[ \min\big( \rho_t \hat A_t,\; \mathrm{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon)\, \hat A_t \big) \Big].$$

DPO — closed-form preference optimization (most popular 2023–24)

Rafailov 2023, arXiv 2305.18290. Reformulates RLHF as classification on the reference policy. Final loss:

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$

IPO, KTO, ORPO — the variants

• IPO (Azar 2023, arXiv 2310.12036) — replaces log-sigmoid with squared loss: $\mathcal{L}_{\text{IPO}} = \mathbb{E}\big[(h_\theta(x, y_w, y_l) - \tfrac{1}{2\tau})^2\big]$ where $h_\theta$ is the policy-ratio log-difference. Prevents overfitting on deterministic preferences ($p \to 1$ pulls the log-sigmoid loss to $-\infty$).
• KTO (Ethayarajh 2024, arXiv 2402.01306) — Kahneman-Tversky Optimization. Doesn't require pairwise data — only binary "good"/"bad" labels per response. Models prospect-theoretic utility with a value function $v(z) = \log\sigma(\beta z)$ for gains, $\lambda \log\sigma(-\beta z)$ for losses (loss-aversion factor $\lambda > 1$). Easier data collection.
• ORPO (Hong 2024, arXiv 2403.07691) — combines SFT + odds-ratio preference loss in one step. Skips the SFT stage entirely; the OR term penalizes high-odds rejected responses.
• SimPO (Meng 2024, arXiv 2405.14734) — length-normalized DPO without a reference model: $\beta \cdot (\tfrac{1}{|y_w|}\log\pi_\theta(y_w \mid x) - \tfrac{1}{|y_l|}\log\pi_\theta(y_l \mid x)) - \gamma$. Cheaper memory, comparable quality.

GRPO — the 2024 winner (DeepSeek)

DeepSeekMath, arXiv 2402.03300 — central role in R1 and V4. For each prompt $x$, sample a group of $G$ outputs $\{y_i\}_{i=1}^G$ from the current policy. Compute rewards $\{r_i\}$. The sequence-level advantage for output $i$ is the group $z$-score:

$$A_i = \frac{r_i - \mathrm{mean}(r_1, \dots, r_G)}{\mathrm{std}(r_1, \dots, r_G) + \varepsilon}$$

This scalar advantage is broadcast to every token in trajectory $i$. The objective adds a per-token KL penalty against $\pi_{\text{ref}}$ (estimated via $\mathrm{KL}[\pi_\theta \| \pi_{\text{ref}}] \approx \tfrac{\pi_{\text{ref}}}{\pi_\theta} - \log\tfrac{\pi_{\text{ref}}}{\pi_\theta} - 1$, the unbiased k3 estimator):

$$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G}\sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \Big( \min(\rho_{i,t} A_i,\, \mathrm{clip}(\rho_{i,t}, 1{-}\epsilon, 1{+}\epsilon) A_i) - \beta\, \mathrm{KL}_t \Big) \right]$$

No critic / value model needed — group statistics replace the value baseline. Removes the critic, saving ~30–50% of total training memory vs PPO (depending on whether the value head shares the policy backbone).

RLAIF — the cheap version

Like RLHF but preference labels come from an AI judge (often a stronger model) instead of humans. Bai et al. (2022) and Lee et al. (2023, arXiv 2309.00267) showed RLAIF matches or beats RLHF on helpfulness, harmlessness, and summarization. The labeling distribution is more consistent than humans (less noise) but inherits the judge's biases.

Constitutional AI (Anthropic)

Bai 2022, arXiv 2212.08073. Two phases:

1. SL phase (self-revision) — model produces a response; same model critiques it against a sampled constitutional principle; same model revises. SFT the final model on (prompt $\to$ revised response) pairs.
2. RL phase (RLAIF) — model generates response pairs; another model picks the better one given the constitution; train RM on these AI-generated preferences; PPO as usual.

The constitution itself is a small set of natural-language principles (be helpful, avoid harm, refuse illegal requests, etc.). Subjectivity shifts from crowd workers to the principles you write. Claude 3 / 3.5 / 4 ship updated constitutions; the "Sparrow rules" (DeepMind) and OpenAI's "Model Spec" are analogous artifacts.

RLVR — the idea

Instead of a learned RM, use a programmatic verifier:

• Unit tests for code (pass/fail), with sandboxed execution.
• Exact-match for math (regex extraction, then symbolic comparison via SymPy / Lean kernel).
• Formal proof checker (Lean, Coq, Isabelle) — sparse but unfakeable.
• Constraint satisfaction (instruction-following with rule grammars; Tülu 3's IF mode).

The verifier is the ground truth on verifiable tasks. The "RL revolution of 2024–26" is largely RLVR-driven. Used by Tülu 3/4, DeepSeek R1/V4, OpenAI o1/o3, Anthropic's reasoning-mode Claudes, Google's Gemini Reasoner.

OpenAI o-series, Claude reasoning mode, Gemini Reasoner

All trained with large-scale RLVR on reasoning traces. The model learns to produce long chains of thought featuring backtracking, self-verification, plan-then-execute, and alternative approaches. Test-time compute (reasoning length) becomes a real monotone knob — Snell et al. (arXiv 2408.03314) and the o1 system card both report power-law scaling of accuracy in $\log(\text{thinking tokens})$.

Compressed Sparse Attention (CSA)

Two-stage approximation of full attention designed for long context. Let $L$ be sequence length and $m = 4$ the stage-A compression factor.

Stage A — softmax-gated pooling. Group tokens into blocks of $m$. For each block $b$ produce a compressed key/value $\tilde k_b, \tilde v_b$ via a gated pool:

$$g_{b,j} = \frac{\exp(w^\top k_{b,j})}{\sum_{j'=1}^{m} \exp(w^\top k_{b,j'})}, \quad \tilde k_b = \sum_{j=1}^m g_{b,j} k_{b,j}, \quad \tilde v_b = \sum_{j=1}^m g_{b,j} v_{b,j}.$$

This reduces the KV sequence from $L$ to $L/m$ entries.

Stage B — Lightning Indexer top-$k$. A separate small attention head (64 heads $\times$ $d_h = 128$, FP32 scores) computes block-level scores $s_b = \mathrm{softmax}(q^\top \tilde k_b)$ and picks the top-1024 blocks for each query. Final attention is computed only against the selected blocks. Total compute is $O(L \cdot k)$ with $k$ fixed at 1024 regardless of $L$.

Heavily Compressed Attention (HCA)

Same idea but $m' = 128$ — aggressive compression — followed by dense attention over the resulting short sequence ($L/128$). Used in every other layer so the model retains a global-mixing pathway at near-zero cost.

Manifold-Constrained Hyper-Connections (mHC)

Generalizes residual connections to multiple parallel streams. Let $X_l \in \mathbb{R}^{n \times d \times s}$ stack $s = 4$ residual streams at layer $l$. The update rule is:

$$X_{l+1} = B_l\, X_l \; + \; C_l\, F_l(A_l\, X_l)$$

where $A_l, B_l, C_l \in \mathbb{R}^{s \times s}$ are mixing matrices and $F_l$ is the block (attention or FFN). The key constraint: $B_l$ is projected onto the Birkhoff polytope of doubly stochastic matrices via 20 iterations of Sinkhorn-Knopp normalization after each step. This keeps the residual streams on a stable manifold, preventing the rank-collapse pathology seen in earlier multi-stream residual variants (e.g., NeuTRENO).

Muon optimizer in production

Muon (Jordan et al. 2024, blog; Liu et al. arXiv 2502.16982) replaces SGD-momentum updates $M_t$ with their orthogonalized versions $\mathrm{Orth}(M_t)$, computed as the polar factor $U V^\top$ of the SVD $M_t = U \Sigma V^\top$. Computed via Newton-Schulz iteration on $X \leftarrow X(3I - X^\top X X^\top)/2$-style polynomials applied to a normalized $M_t$.

V4 uses a hybrid 8+2 Newton-Schulz recipe: 8 iterations with the aggressive coefficient set $(3.4445, -4.7750, 2.0315)$ for rapid convergence, followed by 2 iterations of $(2, -1.5, 0.5)$ for numerical polish near the orthogonal manifold. Applied to all 2D weight matrices (attention projections, FFN, expert weights); embeddings and norms use AdamW. Wall-clock overhead $\sim 5\%$, sample efficiency gain $\sim 1.4\times$ vs AdamW at matched compute.

Mixed-precision recipe

Post-training — cultivation then consolidation

1. Cultivation. Independently fine-tune capability-specific experts from the same base: math expert, code expert, reasoning expert, multilingual expert, safety expert. Each uses domain-specific SFT + RLVR.
2. Consolidation. Use on-policy distillation (Ch. 9) to fold all cultivated experts into a single student — student samples trajectories, the relevant expert provides per-token reverse-KL targets. The student inherits each expert's strengths without their idiosyncrasies.
3. Final GRPO with verifiable rewards on a mixed math/code/IF prompt distribution. Mixed preference rewards for safety and helpfulness.
4. Tool-call token: |DSML| wraps an XML schema. Reasoning traces are preserved across tool boundaries (the model sees its prior CoT after a tool call) but discarded across non-tool conversational turns — prevents context bloat.

V4-Pro benchmark line

R1-Zero / R1 — the predecessor (for context)

DeepSeek R1 (Jan 2025, arXiv 2501.12948) showed that pure RL with verifiable rewards on a strong base elicits reasoning — no SFT required (R1-Zero). The full R1 wrapped that with cold-start SFT, rejection-sampled SFT, and a final mixed-reward RL. Distilled smaller models (Qwen-7B/14B/32B, Llama-8B/70B) inherited reasoning cheaply. V4 supersedes R1 but the R1 4-stage pipeline is still the right "first pass" mental model.

What "interaction model" means

A traditional chatbot is a sequence model over discrete turns. An interaction model is a sequence model over continuous time-aligned micro-turns: the model emits and consumes audio/video/text in lockstep at $\sim 5\,\mathrm{Hz}$ frame rate, so latency is a property of tokens-per-second rather than of full responses. The interlocutor and the model share a single rolling timeline.

Encoder-free early fusion

Older multimodal models bolt on per-modality encoders (Whisper-style for audio, ViT for vision) and pretrained adapters. Interaction Models skip this: every modality is converted to a sequence of embeddings in the same vocabulary space and co-trained from scratch:

• Audio in: dMel embeddings — discretized mel-spectrogram frames (Bai 2024) treated as tokens.
• Video in: 40×40 spatial patches passed through a hierarchical-MLP (hMLP) stem (Touvron 2022, arXiv 2203.04675) — fast, no convolutions.
• Text in: standard BPE.
• Audio out: a flow-matching head (Lipman 2022, arXiv 2210.02747) decodes continuous mel-spectrogram frames; vocoder rendering off-trunk.
• Text out: standard causal LM head.

Co-training from scratch is the central bet: late fusion preserves modality boundaries; early fusion lets the trunk learn cross-modal correspondences directly. The cost is data-mix careful — the team reports needing $\sim 8\%$ pure-text "anchor" data to avoid text-quality regression.

The Background Model

The interaction trunk has to emit at frame-rate, so it can't think for 30 seconds about a tool call. Instead, a separate Background Model shares the same context window and runs async — when the interaction trunk emits a "delegation" token, the background model is woken with the live context, deliberates (potentially with tools), and returns a summary that the interaction trunk consumes when ready. This decouples real-time turn-taking from arbitrary deliberation depth.

FD-bench latency

FD-bench (Full-Duplex bench) measures the time from end-of-user-speech to start-of-model-speech under realistic acoustic conditions. TML-Interaction-Small: $\leq 0.4\,\mathrm{s}$. Gemini-3.1-flash-live: $0.57\,\mathrm{s}$. GPT-realtime-2.0: $1.18\,\mathrm{s}$. The win comes from (1) encoder-free token stream (no Whisper encode latency), (2) MoE-A12B small active footprint, (3) speculative streaming of mel frames into the flow head.

The four flavors of distillation

Why reverse KL is "unhackable"

The student loss is:

$$\mathcal{L}_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)} \big[ \mathrm{KL}\big( \pi_\theta(\cdot \mid x, y_{

Reverse KL $\mathrm{KL}(\pi_\theta \| \pi_{\text{teacher}}) = \sum_y \pi_\theta(y) \log \tfrac{\pi_\theta(y)}{\pi_{\text{teacher}}(y)}$ is mode-seeking: it strongly penalizes any $y$ where $\pi_\theta(y) > 0$ but $\pi_{\text{teacher}}(y) \approx 0$. The student cannot create probability mass on "off-distribution" outputs without being immediately punished — there's no Goodhart gap to exploit, unlike a learned RM where mass on a never-seen response is unconstrained.

Forward KL $\mathrm{KL}(\pi_{\text{teacher}} \| \pi_\theta)$ is mean-seeking — the student spreads mass to cover all teacher modes. Mean-seeking is what classical soft-label distillation does, and it tolerates the student putting mass anywhere as long as it also covers the teacher's modes. Reverse KL forbids this.

Compute numbers

• AIME'24: 70% accuracy at 9–30× lower compute vs GRPO from the same base.
• $7\text{–}10\times$ fewer gradient steps to match an RL-trained checkpoint.
• $50\text{–}100\times$ less total compute (steps × cost-per-step).
• The student trajectories cost almost nothing — only the teacher logits need to be obtained, and that's amortized.

Where it sits in the pipeline

The IBM mid-training study (Ch. 2) places on-policy distillation between SFT and final GRPO. The DeepSeek V4 consolidation step (Ch. 7) is on-policy distillation from multiple capability-specific teachers into one student. Anthropic's "guided RL" methodology in recent Claudes is reported to use a related technique.

The update rule

For a weight $W \in \mathbb{R}^{d \times k}$, LoRA (Hu 2021, arXiv 2106.09685) factorizes the update as a rank-$r$ product:

$$W' = W + \frac{\alpha}{r} B A, \quad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d, k).$$

Only $A, B$ are trained. The $\alpha/r$ scaling decouples learning rate from rank — increasing $r$ doesn't require re-tuning the LR (this is the "LoRA+" / scaling-stable parameterization).

The two preconditions

1. All layers. Apply LoRA to every linear: attention $W_Q, W_K, W_V, W_O$, FFN gate/up/down, MoE expert weights, embeddings if you can afford it. Attention-only LoRA leaves $\sim 70\%$ of parameters frozen and silently underfits.
2. Capacity floor. Trainable parameters must exceed task information content:
   • SFT: $\sim 1$ bit per response token. A 10M-token SFT set needs $\sim 10\,\mathrm{Mbits}$ of trainable params $\to$ $\sim 1.25\,\mathrm{M}$ params $\to$ $r \approx 4\text{–}8$ for a 7B model on all layers.
   • RL: $\leq \log B$ bits per episode with group size $B$. Far less. Rank 1 often suffices for policy gradients.

The optimal LR rule

Empirically across model sizes 1B–70B, tasks, and ranks $r \in \{1, \dots, 256\}$, the optimal LoRA LR sits at $\approx 10\times$ the optimal FullFT LR and is nearly rank-independent when the $\alpha/r$ scaling is used. This contradicts the early-2024 folklore that low-rank LoRA needs higher LR than high-rank LoRA.

Compute and quality

• LoRA FLOPs $\approx \tfrac{2}{3}$ FullFT FLOPs (the $BA$ product is cheap; you skip $W$'s backward).
• For supervised fine-tuning at the capacity floor: matches FullFT.
• For policy gradients (GRPO / REINFORCE): matches FullFT even at rank 1, because the gradient signal is much lower-rank than the weight update space.
• For continued pretraining on $>$10B new tokens: does not match — capacity floor breached.

QAT — train through the quantizer

Insert fake quantization (round-to-low-precision then back) in the forward pass; use the straight-through estimator on the backward — gradient passes through rounding as identity. Train so the model is robust to quantization errors at inference. Used to ship INT4 / NVFP4 inference checkpoints. The forward quantizer at element $x$ with scale $s$:

$$\tilde x = s \cdot Q\!\left(\frac{x}{s}\right), \quad \frac{\partial \tilde x}{\partial x} \approx 1 \;\;(\text{STE}).$$

FP8 training recap (still the workhorse for most labs)

• E4M3 (4-bit exponent, 3-bit mantissa) — forward and weights.
• E5M2 (5-bit exponent, 2-bit mantissa, larger range) — gradients.
• FP32 master weights for optimizer state.

DeepSeek V3's reference recipe (arXiv 2412.19437): per-tile scaling (1×128 activations / 128×128 weights), online scale computation, FP32 accumulation promotion every 128 elements, BF16 fallback for embeddings/output/norms.

MXFP4 / NVFP4 — the FP4 era

OCP Microscaling 2024 standard: block of 32 elements share a single 8-bit (E8M0) power-of-2 scale. MXFP8, MXFP6, MXFP4 are the formats. NVIDIA's NVFP4 variant uses an FP8 (E4M3) per-block scale plus an FP32 per-tensor scale — strictly higher accuracy than MXFP4 for similar memory.

Quartet, Quartet II, Four-Over-Six

• Quartet (Castro 2025, arXiv 2505.14669) — first BF16-parity FP4 pretraining; hardware-accelerated on Blackwell B200.
• Quartet II (arXiv 2601.22813) — introduces MS-EDEN, an unbiased mean-shift quantizer with $>2\times$ lower mean-squared error than stochastic rounding at the same compute.
• Four-Over-Six (arXiv 2512.02010) — adaptive block scaling: each block independently chooses between FP4 and FP6 based on its dynamic range. Spends $\sim 17\%$ of blocks on FP6, recovers FP6 quality at $\sim 1.1\times$ FP4 cost.
• NVFP4 pretraining (NVIDIA, arXiv 2509.25149) — 12B hybrid Mamba-Transformer on 10T tokens in NVFP4 with all five techniques above.

DiLoCo recap

DiLoCo (Douillard 2023, arXiv 2311.08105) is "local SGD" for LLMs. Each worker (a full datacenter) runs $H$ inner steps with AdamW on local data, then performs an outer step where the workers' parameter deltas are averaged. The outer step uses an outer optimizer (typically Nesterov momentum). Communication frequency drops from once-per-step to once-per-$H$-steps — orders of magnitude less bandwidth.

$$\theta^{(t+1)} = \theta^{(t)} - \eta_{\text{outer}} \cdot \mathrm{Outer}\!\left( \frac{1}{K} \sum_{k=1}^K \big( \theta_k^{(t,H)} - \theta^{(t)} \big) \right)$$

What "decoupled" adds

• Async learners. Workers do not wait for each other's outer step. Stale deltas are accepted up to a configurable staleness bound.
• CPU-side parameter-fragment synchronizer. A separate CPU service shards the parameter space, accumulates incoming deltas, and applies the outer update incrementally. Workers pull updated fragments asynchronously. This removes the head-of-line blocking that synchronous DiLoCo suffered when one region's link spiked.
• Minimum-quorum + adaptive grace window. Outer step proceeds when $\geq M$ of $K$ workers have submitted (typically $M = \lceil 0.6 K \rceil$); remaining workers get an adaptive grace window before being declared stale. The grace window adapts to recent link-latency statistics.
• Goodput-aware scheduling. Slow regions get fewer outer-step votes; the system trades wall-clock for participation.

Headline numbers

12B model, 4 US regions (us-east-1, us-east-4, us-west-1, us-central-1), 2–5 Gbps inter-region links, $H = 500$, $K = 4$ workers each $\sim$2048 H200 GPUs. Result: $20\times$ wall-clock speedup vs synchronous baseline at the same total compute, and 88% goodput with simulated link failures injected at typical cloud-provider rates.

The strategic implication: training a frontier-scale model no longer requires a single $100\,\mathrm{k}$-GPU datacenter with InfiniBand. You can stitch four 25k-GPU regions together over the public internet — a fundamentally different supply-chain story.

µP in one sentence

Under the maximal-update parameterization, the optimal learning rate for each layer scales as $\eta_l \propto 1/\mathrm{fan\_in}_l$, and the same nominal $\eta$ produces feature updates of $\Theta(1)$ at every width. Tune $\eta$ at width $d = 256$, deploy at $d = 16384$, no re-tune. Without µP, optimal LR depends nontrivially on width and you re-tune at every scale (Llama 3's HP sweep cost ~5% of total training FLOPs).

u-µP — across precision

µP transfers across width but not across precision. u-µP composes µP with unit-scaling (Blake 2023, arXiv 2303.11257), which constrains every tensor's pre-quantization variance to $1$. Result: HPs tuned in BF16 on a proxy transfer to FP8 / NVFP4 at frontier scale. Critical when you can't afford an FP8 HP sweep.

GQA-µP — surviving the stack

The original µP derivation assumes vanilla multi-head attention and no parameter sharding. GQA breaks the fan-in counting (KV heads are shared); FSDP shards the parameters and changes the effective initialization geometry. GQA-µP rederives the scaling rules for the actual layer stack used at the frontier and validates HP transfer across $d_{\text{model}} \in [256, 16384]$ and across 1- to 64-way FSDP.

MoE routing in one line

For input $x$, router logits $g(x) = W_g x \in \mathbb{R}^E$, top-$K$ selection produces a sparse gate $G(x) \in \mathbb{R}^E$ with $K$ nonzero entries. Output is $\sum_e G(x)_e \cdot \text{Expert}_e(x)$. Top-$K$ over a uniform initial $W_g$ collapses to a few experts within hundreds of steps without a balancing mechanism.

The old way — auxiliary loss

Shazeer's original load-balance loss (and the importance-loss variant) adds:

$$\mathcal{L}_{\text{aux}} = E \cdot \sum_{e=1}^E f_e \cdot p_e, \quad f_e = \frac{1}{B}\sum_{i} \mathbb{1}[e \in \text{top-}K(x_i)], \quad p_e = \frac{1}{B}\sum_{i} \mathrm{softmax}(g(x_i))_e.$$

This pushes load toward uniform but its gradient flows through $g$ and therefore through $W_g$, in opposition to the LM-loss gradient. Quality degrades as $\beta_{\text{aux}}$ grows; underdamping leaves load imbalanced.

Auxiliary-loss-free — the new way

Maintain a per-expert bias $b_e$ added to logits only at routing decision time:

$$\hat g(x)_e = g(x)_e + b_e, \quad \text{top-}K(\hat g(x)).$$

After each batch, update:

$$b_e \leftarrow b_e + \gamma \cdot \mathrm{sign}\big( \bar f - f_e \big)$$

where $\bar f$ is the target average load (typically $K/E$) and $\gamma$ is a small step size (e.g., $10^{-3}$). The bias is not a parameter that receives backprop gradient — it's updated like an EMA of utilization. Result: no interference with the LM loss, well-balanced load, no quality regression.

Theory: Wang & Sun (arXiv 2512.03915) show this is a stochastic-approximation algorithm targeting balanced utilization, and prove convergence under mild conditions on the routing-logit distribution.

Position interpolation family

RoPE rotates query/key by angle $\theta_d \cdot m$ at position $m$, dimension $d$, base $\theta_d = b^{-2d/D}$ with $b = 10000$ (typical). To extend from $L_{\text{train}}$ to $L_{\text{target}}$:

• Position Interpolation (Chen 2023, arXiv 2306.15595) — rescale positions $m \to m \cdot L_{\text{train}}/L_{\text{target}}$. Continue training briefly. Loses some short-context quality.
• NTK-aware scaling — instead of uniform position scaling, scale the RoPE base: $b \to b \cdot s^{D/(D-2)}$ with $s = L_{\text{target}}/L_{\text{train}}$. Higher $b$ $\to$ longer wavelengths $\to$ less position aliasing at long range.
• YaRN (Peng 2023, arXiv 2309.00071) — frequency-categorized interpolation + attention-temperature correction: high-frequency dims (small $\theta$) extrapolate; low-frequency dims interpolate; in between blend. Used to extend Llama from 4k $\to$ 128k.
• LongRoPE2 (arXiv 2502.20082) — per-dimension search of optimal scaling factors via evolutionary algorithm; extends to 2M context.

Ring & striped attention — sequence-parallel for the actual training

• Ring Attention (Liu 2023, arXiv 2310.01889) — partition sequence across $N$ GPUs. Each GPU holds $Q, K, V$ for its chunk. Pass $K, V$ around the ring, accumulating attention. Combined with FlashAttention's block-wise softmax, you can train on 1M+ context with no quadratic memory blow-up.
• Striped Attention — load-balanced ring (causal mask creates uneven work; striping fixes it).
• Tree Attention (Liu 2024, arXiv 2408.04093) — replaces ring with tree-reduce for lower latency on large $N$.

Native long-context architectures

V4's CSA + HCA (Ch. 7), Gemini's Titans/Mneme-style memory layers, and Mamba/SSM hybrids all attack the same problem from the architecture side: keep effective attention cost sub-quadratic so 1M+ is a first-class regime, not a stretch goal. The trade-off is exact-retrieval fidelity (full attention) vs scaling (compressed/sparse). 2026 frontier models accept the trade-off; the loss on needle-style retrieval is tolerable given the latency win.

Long-context evaluation

Standard suite: needle-in-haystack (planted fact — necessary but not sufficient), RULER (Hsieh 2024, multi-needle + multi-hop), LongBench v2, BABILong, multi-doc QA, InfiniteBench.

Curriculum / rejection sampling / best-of-N (bonus toolkit)

• Rejection sampling fine-tuning (RFT) — sample many completions, keep only correct (verified or RM-scored), SFT on those. Llama 3 RLHF had a rejection sampling stage.
• Best-of-N — at inference, sample $N$, pick highest-RM-scoring. Strong baseline; often matches RL-tuned model up to $N = 64$.
• Self-distillation — train student on outputs of a (possibly identical-arch) teacher. Born-Again Networks; iterative self-distillation sometimes improves quality.
• Curriculum learning — order data easy $\to$ hard. In LLMs, results mixed; usually domain-mixing changes (mid-training, annealing) matter more.

LLM training quiz — readiness check

1. Walk through Llama 3's training pipeline.
   Show answer

   15T pretraining tokens with quality classifier on web; MinHash dedup; phased annealing in last 40B tokens. Then SFT + rejection sampling + DPO. Architecture: GQA + RoPE + RMSNorm + SwiGLU. Scaling: 8B, 70B, 405B variants. Multimodal added in 3.2.

2. Why GRPO over PPO?
   Show answer

   No critic/value model → saves ~30–50% of total training memory (depends on value-head sharing); group-relative advantage is well-conditioned for verifiable rewards; simpler to scale. Tradeoff: only sequence-level credit assignment.

3. How would you train a reasoning model from scratch?
   Show answer

   Strong base → RLVR with GRPO + verifiable rewards (math, code) → SFT on rejection-sampled good traces → final RL with mixed verifiable + preference rewards. Distill into smaller dense models.

4. How does Constitutional AI differ from RLHF?
   Show answer

   Preferences come from an AI judge applying explicit principles, not humans. SL phase: model self-critiques + revises per principles. RL phase: AI-generated preference pairs train RM, then PPO. Reduces human labeling cost; shifts subjectivity to constitution.

5. What is mode collapse in DPO?
   Show answer

   DPO can decrease likelihood of both chosen and rejected (only the gap matters); policy may drift far from reference. Mitigations: stronger β, IPO (squared loss), mix in SFT loss, conservative sampling.

6. Walk through DeepSeek V3's FP8 recipe.
   Show answer

   E4M3 forward + weights, E5M2 backward. Per-tile (1×128 activations / 128×128 weights) scaling. FP32 master weights. FP32 partial-sum promotion every 128 elements. BF16 fallback for embeddings, output head, normalization. ~2× speedup vs BF16 with no quality loss.

7. Explain Chinchilla and why models today violate it.
   Show answer

   Chinchilla: D ≈ 20·N is compute-optimal during training. But inference cost dominates total cost when serving for years → over-train smaller models past Chinchilla optimum (Llama 3 8B on 15T tokens = 1875 tokens/param) for cheaper per-query inference.

8. What's the closed-form derivation of DPO?
   Show answer

   Start from KL-constrained RL: max E[r] − β KL(π||π_ref). Closed-form optimum: π* = (1/Z) π_ref · exp(r/β). Solve for r: r = β log(π*/π_ref) + β log Z. Plug into Bradley-Terry preference; Z(x) cancels. Result: DPO loss equals NLL of preferences under the implicit reward model induced by the policy.

9. What's RFT (rejection sampling fine-tuning)?
   Show answer

   Sample many completions per prompt; keep only correct (verified or RM-scored); SFT on those. Llama 3 RLHF used this. Cheaper than RL; often nearly as good. Used as a stage in DeepSeek R1's pipeline.

10. What is RLVR and what's its main advantage over RLHF?
    Show answer

    RL with verifiable rewards: programmatic verifier (unit tests, math grader, formal proof checker) instead of learned RM. No reward hacking on the verifier signal. Drives reasoning-model training (o-series, R1, Tülu 3).

11. Walk through DeepSeek R1's pipeline (4 stages).
    Show answer

    (1) Cold-start SFT: small set of curated reasoning traces with desired format. (2) RL with GRPO + verifiable rewards. (3) Rejection-sampling SFT: 600k math/code from stage-2 + 200k general. (4) Final RL: mixed verifiable + preference rewards. Distill into dense models.

12. What is MinHash LSH for dedup?
    Show answer

    Compute MinHash signatures over n-gram shingles (5-grams typical). Bucket into LSH bands; candidate pairs share at least one band. Verify with Jaccard. For 5T docs, typically 10 bands × 9 hashes (controls precision/recall). Used by DeepSeek, Llama, FineWeb.

13. What is SemDeDup?
    Show answer

    Cluster embeddings with k-means; dedup within clusters by cosine similarity. Catches near-duplicates that MinHash misses (paraphrases, translations, reformatting). Used as a complement to MinHash.

14. Why mid-training before SFT?
    Show answer

    A phase between pretraining and SFT: continue causal LM but heavy upweight on reasoning, math, code, possibly synthetic CoT. Combined with WSD decay phase. Model becomes "reasoning-ready" without yet being instruction-tuned. DeepSeek V3 used this.

15. What's the LIMA hypothesis?
    Show answer

    (arxiv 2305.11206) "Less is more for alignment" — diversity > scale beyond ~10k–100k SFT examples. The model's capabilities come from pretraining; SFT only teaches format/instruction-following. Recent work (Tülu 3) shows up to 1M+ helps when domains are diverse.

16. What is REINFORCE and why mention it now?
    Show answer

    Original policy gradient: ∇L = −E[log π(a|s) · A]. PPO adds clipping + value baseline. RLOO / REINFORCE++ (Ahmadian 2024) skip the critic (like GRPO) but use leave-one-out or EMA baselines. Coming back into fashion for LLM RL.

17. What is KTO?
    Show answer

    Kahneman-Tversky Optimization (Ethayarajh 2024). Doesn't require pairwise data — only binary "good"/"bad" labels per response. Models prospect-theoretic utility. Easier data collection.

18. What's catastrophic forgetting and how do you mitigate?
    Show answer

    Continued training on new data degrades performance on old data. Mitigations: (1) replay (mix old data, 5-30%); (2) much lower LR; (3) PEFT (LoRA — train only adapters); (4) elastic weight consolidation; (5) regularize toward original weights.

19. What's the difference between in-context learning and fine-tuning?
    Show answer

    ICL: no weight updates; few-shot examples in prompt steer behavior. Fast, no infra, but limited context, no persistence. Fine-tuning: weight updates persist; better quality on narrow domain; risks catastrophic forgetting; needs infra. 2026 guidance: try prompt → RAG → SFT/LoRA → full FT → RL in order.

20. What's reward hacking in reasoning models?
    Show answer

    Model gaming the reward signal: tampered CoT (correct answer with non-causal reasoning); verifier exploitation (matches the regex but reasoning is wrong); length hacking. Mitigations: process rewards (PRM), multiple verifiers, behavioral evals catching CoT-answer mismatch.