ML theory — the core fundamentals

The decomposition — derive it on the whiteboard

Let y = f(x) + ε with ε ~ N(0, σ²) independent of D. Train predictor f̂(x; D) on dataset D. Expected squared error at point x:

E_D[(y − f̂(x; D))²]
  = E_D[(f(x) + ε − f̂(x; D))²]                                  # substitute
  = E[ε²] + E_D[(f(x) − f̂(x; D))²]                              # cross terms vanish (ε ⊥ D)
  = σ²
  + E_D[(f(x) − E_D[f̂(x; D)])²]                                 # = bias² (squared dist of mean prediction)
  + E_D[(f̂(x; D) − E_D[f̂(x; D)])²]                              # = variance
  ─────────────────────────────────
  = σ²        +    bias²(f̂; x)    +    var(f̂; x)
   irreducible      (model wrong on average)   (model unstable across datasets)

What bias and variance buy you in practice

Why bagging vs boosting attack different terms

• Bagging (random forest): variance ↓↓, bias ≈ same. Average decorrelated trees.
• Boosting (GBM): bias ↓↓, variance can ↑ if not regularized. Each tree fixes prior residuals.
• Adding features: usually ↓ bias, can ↑ variance if features are noisy.

Where deep nets break the textbook U-curve

Classical view: model capacity ↑ → bias ↓, variance ↑ → U-shaped test error.

The base rule and what convexity buys

For loss L(θ): θ ← θ − η ∇L(θ). Convergence rates depend entirely on the problem class:

The condition number κ = β/μ controls speed; ill-conditioned problems crawl.

Why SGD generalizes better than full-batch

Empirically (Keskar 2017): small-batch SGD finds flatter minima than large-batch, and flatter minima generalize. Two main theoretical lenses:

• Implicit regularization: SGD noise covariance ∝ Hessian-related quantities in low-loss regions, biasing the trajectory toward low-curvature (flat) minima.
• Critical batch size (McCandlish 2018): B_crit ∝ tr(Σ)/||g||² where Σ is per-sample gradient covariance. Past B_crit, the linear-LR scaling rule (Goyal 2017) breaks.

Momentum, Nesterov, Newton, quasi-Newton

• Momentum (Polyak heavy-ball): v ← μv + ∇L; θ ← θ − ηv. Variance reduction; smooths noise.
• Nesterov: gradient at lookahead point θ − ημv. Theoretically optimal first-order rate O(1/k²) for convex smooth.
• Newton: θ ← θ − H⁻¹ ∇L. Quadratic convergence near optimum. H is N×N — infeasible for N ≫ 1000.
• Quasi-Newton (BFGS, L-BFGS): approximate H⁻¹ from gradient history. Fails in stochastic settings.
• Adam: diagonal preconditioner using grad².
• Shampoo: Kronecker-factored preconditioner.
• K-FAC: block-diagonal Fisher approximation.

Adam, AdamW, and the L2 bug

Adam maintains EMAs of grad and grad². Update:

m ← β₁m + (1−β₁) g          # 1st moment EMA
v ← β₂v + (1−β₂) g²         # 2nd moment EMA
m̂ = m / (1 − β₁^t)          # bias correction
v̂ = v / (1 − β₂^t)
θ ← θ − η · m̂ / (√v̂ + ε)

The bug: standard L2 in Adam adds λθ to the gradient, which then gets divided by √v̂ — so high-variance params receive less decay. AdamW (Loshchilov 2019) decouples: apply θ ← θ − η(m̂/√v̂ + λθ) instead. AdamW is the modern default for transformers.

Why conditioning is the silent optimizer

The Hessian eigenvalue spread κ = λ_max / λ_min determines GD speed. In deep nets, κ can exceed 10⁶. Three tricks fight bad conditioning:

1. Feature scaling / normalization: gives input dims similar variance → conditions the Hessian's input block.
2. Weight init (He / Xavier): scales weights so activations and gradients stay comparable across layers — prevents conditioning collapse at depth.
3. Batch / Layer norm: re-scales activations → smoother loss landscape (Santurkar 2018). The original "internal covariate shift" claim is largely wrong; the real mechanism is Lipschitz smoothing.

Sigmoid's vanishing gradient — the actual numbers

Sigmoid: σ(z) = 1/(1+e⁻ᶻ); σ'(z) = σ(z)(1−σ(z)). Maximum at z=0: σ'(0) = 0.25.

Backprop through L sigmoid layers multiplies L derivatives. Even at the maximum:

Tanh is slightly better (tanh'(0) = 1.0) but still saturates outside a narrow band.

ReLU's properties — what changed

• Derivative: 1 if z > 0, 0 otherwise. No saturation in the positive regime → no shrinking gradient through depth.
• Sparsity: ~half the units are 0 at init → sparse activation, better feature selectivity, cheaper compute.
• Cost: cheap (a max op).
• Why not identity: without nonlinearity, stacked linear layers collapse to one linear layer — no expressivity.

Why GELU / SwiGLU now in transformers

ReLU's hard cutoff loses info at z near 0. GELU is smooth: x · Φ(x) — small negative inputs still pass a small signal. SwiGLU (Shazeer 2020) adds gating: (x W₁) · σ(x W₂). Empirically improves quality at fixed FLOPs.

None of this would scale without residual connections. Residuals give every layer an "identity path" so the gradient never has to multiply through the activation derivative — see chapter 14.

L2 (ridge) — the round constraint

Penalty: L = L_0 + λ ||θ||₂² = L_0 + λ Σ θᵢ².

• Constraint geometry: L2 ball — circle in 2D, sphere in N-D. Smooth.
• Lagrangian solution: optimum is where the loss contour first touches the ball — a generic point, no coordinate zeros out.
• Closed form for linear regression: (XᵀX + λI)⁻¹ Xᵀy. Always invertible (even when XᵀX is singular).
• Bayesian: prior θ ~ N(0, σ²/λ · I) → MAP equals ridge.
• Gradient: 2λθ — proportional to weight; pulls all weights toward 0 uniformly, never reaches 0.

L1 (lasso) — the diamond constraint

Penalty: L = L_0 + λ ||θ||₁ = L_0 + λ Σ |θᵢ|.

• Constraint geometry: L1 ball — diamond in 2D, hyperoctahedron in N-D. Corners aligned with axes.
• Optimum often touches a corner → some coordinates exactly 0 → sparse solution.
• No closed form; solved with proximal gradient (ISTA, FISTA) or coordinate descent.
• Bayesian: prior θ ~ Laplace(0, b) → MAP equals lasso.
• Gradient (subgradient at 0): λ · sign(θ) — constant pull; stops once at 0.

Side by side

Why L1 is sparse — visual + math

Visual: imagine the elliptical loss contour tightening toward its minimum. With L2 (round constraint) the first touch is generic. With L1 (diamond) the first touch is most often at a vertex — and vertices have all-but-one coordinate equal to zero.

Math: at a coordinate where |∂L_0/∂θᵢ| < λ, the L1 subgradient zeros that coordinate. L2 only shrinks (multiplicative); L1 thresholds (the soft-threshold operator sign(θ) · max(|θ|−λ, 0)).

Elastic net — the practical compromise

α L1 + (1−α) L2. Combines sparsity with L2's grouping (correlated features both retained or both dropped together). glmnet (Friedman, Hastie, Tibshirani 2010) is the canonical implementation.

Why weight decay works in deep learning (different reason)

In deep nets, weight decay (≈ L2) is critical for generalization, but for different reasons than classical statistics:

• Caps the effective scale of weight matrices → bounds Lipschitz constant → smoother decision boundary.
• Interacts with batch norm: BN makes scale invariant; weight decay then controls the angular direction of weights (effectively projects to unit norm).
• AdamW decoupling: standard L2 in Adam scales decay by 1/√v̂ — small decay for high-variance params, undesirable. AdamW separates decay from gradient step.

MLE — the framework

Choose θ to maximize P(D|θ) = ∏ P(xᵢ|θ). Equivalently, minimize −log P(D|θ) = −Σ log P(xᵢ|θ). This is the loss.

Common losses are NLLs in disguise

Why cross-entropy beats MSE for classification

• Probabilistic correctness: CE matches the Bernoulli/Categorical likelihood, MSE does not.
• Gradient signal: CE gradient w.r.t. pre-softmax logits is p̂ − y — clean and bounded. MSE gradient through softmax involves an extra σ'(z) factor that vanishes when σ saturates → slow learning.
• Calibration: CE produces (more) calibrated probabilities; MSE-trained classifiers are systematically miscalibrated.

MAP — adding a prior

argmax P(θ|D) = argmax P(D|θ) P(θ) by Bayes. Adding the prior is adding a regularizer:

Full Bayes — integrating out θ

P(θ|D) ∝ P(D|θ) P(θ); predict by P(y|x, D) = ∫ P(y|x, θ) P(θ|D) dθ. Almost never tractable. Approximations: MCMC (HMC, NUTS), variational inference, Laplace approximation.

Conjugate priors — still asked

Entropy and cross-entropy

KL divergence — the workhorse

KL(p || q) = Σ p(x) log(p(x)/q(x)) = H(p, q) − H(p). Asymmetric; not a metric. KL ≥ 0 (Gibbs); = 0 iff p = q. Used everywhere: VAE ELBO, RLHF KL penalty, knowledge distillation.

Why the asymmetry matters — mode-covering vs mode-seeking

Mutual information and InfoNCE

I(X; Y) = KL(P(X,Y) || P(X)P(Y)) = H(X) − H(X|Y) = H(Y) − H(Y|X). Reduction in uncertainty about X given Y. InfoNCE (van den Oord 2018) is a tractable lower bound on I(X;Y); it's the loss behind SimCLR, CLIP, and most modern contrastive learning.

Jensen-Shannon divergence

JS(p, q) = ½ KL(p || m) + ½ KL(q || m) where m = ½(p+q). Symmetric; bounded in [0, log 2]; used in original GAN.

Why KL in RLHF?

The KL penalty r − β · KL(π || π_ref) prevents the policy from drifting too far from the reference (SFT model). Without it, RL mode-collapses onto reward-hacking patterns. β trades off reward maximization vs staying-close-to-reference.

PCA via SVD

Center X (subtract column means). SVD: X = U Σ Vᵀ. Then:

• Columns of V are principal directions (eigenvectors of XᵀX).
• Σ²/(n−1) are eigenvalues = variances along principal directions.
• Projection onto top-k components: X V[:, :k].
• Equivalent: U Σ = principal scores; truncate.

SVD identities to know cold

• Eckart-Young: truncated SVD = best rank-k approximation in Frobenius norm (and in the spectral norm).
• Pseudoinverse: X⁺ = V Σ⁺ Uᵀ where Σ⁺ inverts non-zero singular values.
• Condition number: σ_max / σ_min. Ill-conditioned matrices are hard to invert / optimize.

Why PCA matters at interviews

• Dimensionality reduction (visualization, compression).
• Decorrelation of features → conditioning of optimization.
• Eigenfaces, latent semantic analysis, embedding analysis.
• Connection to autoencoders: a linear autoencoder with MSE loss recovers the PCA subspace.

The kernel trick

Map data to high-dim feature space φ(x); train linear model there. Trick: any algorithm that depends on data only through inner products ⟨x, x'⟩ can be "kernelized" by replacing inner products with a kernel K(x, x') = ⟨φ(x), φ(x')⟩ — without ever computing φ explicitly.

Common kernels

• Linear: K(x, x') = xᵀx'
• Polynomial: (xᵀx' + c)^d
• RBF / Gaussian: exp(−γ ||x − x'||²) — corresponds to infinite-dim feature space
• Laplacian: exp(−γ ||x − x'||₁)

SVM derivation — be ready to whiteboard

Maximum-margin classifier. For separable data with labels y ∈ {−1, +1}:

min_{w, b} ½ ||w||²
subject to  y_i (wᵀx_i + b) ≥ 1  for all i

Why ½||w||²? Margin = 2/||w||. Maximize margin ↔ minimize ||w||.

Soft margin (real data is not separable):

min ½||w||² + C Σ ξ_i
s.t. y_i(wᵀx_i + b) ≥ 1 − ξ_i,  ξ_i ≥ 0

The Lagrangian dual is what gets kernelized. Dual variables α_i: w = Σ α_i y_i x_i. Only support vectors (α_i > 0) contribute. Replace x_iᵀ x_j with K(x_i, x_j) in the dual → kernel SVM.

Hinge loss — the SVM in loss form

SVM is equivalent to minimizing max(0, 1 − y · f(x)). Hinge gives 0 loss for correctly classified points outside the margin → only "marginal" examples drive learning. Compare to logistic loss log(1 + exp(−y · f(x))) which has nonzero gradient everywhere.

Decision trees

• Greedy split selection: at each node, pick (feature, threshold) maximizing Gini / entropy / variance reduction.
• Stopping: max depth, min samples leaf, min impurity decrease, max leaves (LightGBM-style).
• Pruning: cost-complexity (alpha-pruning).

Random forest (Breiman 2001)

• Bagging: train T trees on bootstrap samples; aggregate by majority vote / mean.
• Random feature subset per split: decorrelates trees → variance ↓↓.
• Out-of-bag (OOB) error: free CV estimate.

Gradient boosting

Sequentially fit weak learners to the residuals (or, generally, to the negative gradient of the loss). For squared loss, residual = y − ŷ. For other losses, fit pseudo-residuals = −∂L/∂ŷ.

Common interview Qs on trees

• "Why do GBMs beat NNs on tabular?" → Tabular features have axis-aligned structure that trees exploit; NNs need lots of data to learn that. NN catches up at scale (TabPFN, FT-Transformer).
• "Why is LightGBM faster than XGBoost?" → Histogram-based binning, leaf-wise growth (vs level-wise), GOSS + EFB.
• "How does XGBoost handle missing values?" → At each split, missing samples are sent to the side that yields better gain (learned default direction).

EM in two steps

• E-step: compute posterior over latents given current params: q(z) = P(z | x, θ_t).
• M-step: maximize expected complete-data log-likelihood w.r.t. θ: θ_{t+1} = argmax_θ E_q[log P(x, z | θ)].

Guarantee: each iteration monotonically increases the marginal likelihood log P(x | θ). Converges to local optimum.

The three components

1. Random component: y has distribution from exponential family (Gaussian, Bernoulli, Poisson, Gamma, ...).
2. Systematic component: linear predictor η = xᵀβ.
3. Link function: g(μ) = η. Connects E[y] to η.

Why this framework matters

• Unifies many models under one estimation procedure (IRLS — iteratively reweighted least squares).
• Logistic regression's "wisdom" — convex loss, no local minima — comes from GLM theory.
• Calibration story: canonical link guarantees E[y] = link⁻¹(xᵀβ) → predictions are calibrated probabilities.

Measuring calibration

• Reliability diagram: bin predictions into quantiles, plot mean predicted vs mean observed in each bin.
• ECE (Expected Calibration Error): Σ (|B_m|/n) · |acc(B_m) − conf(B_m)|. Sensitive to binning choice.
• Brier score: Σ (p̂ − y)². Decomposes into calibration + sharpness components.
• Calibration slope / intercept: fit logistic on (prediction, label); slope=1, intercept=0 = calibrated.

Recalibration techniques

Why neural nets are miscalibrated

(Guo 2017, "On Calibration of Modern Neural Networks") Modern over-parameterized NNs achieve very low cross-entropy but are overconfident. Hypotheses: capacity-driven overfit on confidence; BN; weight decay shifting output scale. Temperature scaling fixes most of it without retraining.

PAC learning

A hypothesis class H is PAC-learnable if there's an algorithm such that for any ε, δ > 0, with prob ≥ 1−δ over samples, the learned h has error ≤ ε. Sample complexity bounded by VC dimension.

VC dimension

Largest set of points that H can shatter (label in all 2ⁿ ways). VC of linear classifiers in d-dim = d+1. Generalization bound: err_test ≤ err_train + O(√(VC/n)). Vacuous for over-parameterized NNs (VC ~ # params ≫ n).

Rademacher complexity

Tighter than VC for many cases. R_n(H) = E_σ [sup_{h ∈ H} (1/n) Σ σ_i h(x_i)] where σ_i are Rademacher (±1) random vars. Margin-based bounds (Bartlett) scale with weight norms / Lipschitz constants instead of param count → less vacuous for deep nets.

PAC-Bayes (used by Anthropic for SAEs etc.)

Bound expected loss over a posterior Q: E_Q[err_test] ≤ E_Q[err_train] + O(√(KL(Q || P) / n)). P = prior, Q = posterior. Tight when Q is close to P. Dziugaite & Roy 2017 gave non-vacuous PAC-Bayes bounds for MNIST nets — a milestone.

Implicit bias of SGD / NTK / μP

• NTK (Neural Tangent Kernel): in infinite-width limit, NN training behaves like kernel regression with the NTK. Gives a clean theory of generalization in this limit but doesn't capture feature learning.
• μP (Tensor Programs): parameterizations that preserve feature learning at scale; HPs transfer.
• Implicit bias for separable data (Soudry 2018): logistic regression w/ SGD on linearly separable data → max-margin classifier (same as SVM). For NNs, implicit bias is to "low-norm" interpolating solutions.

Why does dropout work?

• Ensemble interpretation: each dropout mask defines a sub-network; training averages over an exponential ensemble. Test-time scaling by (1−p) approximates the ensemble.
• Bayesian interpretation (Gal & Ghahramani 2016): dropout is a variational approximation to a Gaussian process posterior.
• Co-adaptation prevention: forces each unit to be useful regardless of which other units are active.
• Modern LLM pretraining usually uses dropout=0 — we're data-bottlenecked, not capacity-bottlenecked. SFT and small-data fine-tune still use it.

Why does batch norm work?

Original claim (Ioffe & Szegedy 2015): "internal covariate shift" reduction. Newer view (Santurkar 2018, arxiv 1805.11604): BN makes the loss landscape smoother (smaller Lipschitz constant of gradients), allowing larger LRs. The "covariate shift" claim is largely wrong.

• Train/test discrepancy: BN uses batch stats during train, running averages at test → bug source if batch composition differs.
• Doesn't work for very small batches (noisy stats) → GroupNorm / LayerNorm replace it.
• For sequences (transformers), LN is preferred — no batch dependence.

Why does data augmentation work?

• Effective regularization: increases the effective dataset size.
• Encodes invariances directly (translation, rotation, color jitter for vision).
• For LLMs: synthetic data via stronger model = augmentation.

Why does early stopping work?

Equivalent to L2 regularization in the linear/quadratic regime (Yao 2007). Stops training before the model fits noise; an implicit constraint on model complexity via the optimization trajectory.

Why does feature normalization help?

Conditioning. The Hessian's eigenvalues depend on input scales. Normalizing features → Hessian eigenvalues become more similar → larger LR works → faster convergence. Without normalization, GD takes tiny steps in the small-eigenvalue directions.

Why does deep beat shallow?

• Compositional features: depth gives exponentially more expressive function classes for the same param count (Telgarsky 2016).
• Feature reuse: lower layers extract reusable primitives (edges → textures → parts → objects).
• Implicit bias toward low-rank / structured solutions: depth + GD favors hierarchical decompositions.
• Caveat: at fixed compute, very-deep-very-narrow vs moderate-depth-wide is empirically narrow-wins for transformers; inductive bias matters.

Why does the residual connection matter?

Without residuals, gradient through L layers is a product of L Jacobians → either vanishes or explodes. With residuals, gradient has an "identity path" — ∂(x + f(x))/∂x = I + ∂f/∂x — preserving signal. Lets you train 100+ layers (ResNet, transformers).

Why does softmax saturate / how to avoid?

For very large logits, softmax outputs ≈ 1.0 for the max, ≈ 0 for the rest → gradient through the rest vanishes. Fixes: temperature scaling, label smoothing, gradient clipping. In transformers, the √d_k attention scale is exactly to prevent this.

The 50 deep-understanding questions

1. Q. "If your validation loss is much higher than train loss, but test accuracy is fine, what's going on?"
   Show answer

   Almost always the model is overconfident on misclassified validation examples — the cross-entropy loss penalizes confidence × wrongness, while accuracy is binary. The model has learned to be very confident on most examples and is occasionally very wrong; that drives loss up while accuracy stays good. Fix: temperature scaling for calibration if probability matters; otherwise it's not a real problem.

2. Q. "Why isn't ReLU differentiable at 0, and why doesn't that break SGD?"
   Show answer

   ReLU has a kink at 0 — left derivative = 0, right derivative = 1. The probability of a stochastic-gradient update landing exactly at 0 is measure zero. In practice we use either the subgradient (any value in [0,1]) or set the derivative at 0 to 0 by convention. SGD's robustness to non-smoothness comes from finite-precision floating point + stochasticity.

3. Q. "Why does adding L2 regularization improve a model that's not overfitting?"
   Show answer

   L2 doesn't only fight overfitting. It (1) improves Hessian conditioning (adds λI → eigenvalues bounded below by λ), (2) makes training more stable / faster convergence, (3) bounds Lipschitz constant → robustness, (4) for NNs interacting with batch norm, controls effective angular direction. So even at-capacity models benefit.

4. Q. "What's wrong with using accuracy on imbalanced classes?"
   Show answer

   If 99% of samples are negative, a model predicting always-negative gets 99% accuracy with zero recall. Use precision/recall, F1, AUC-ROC (threshold-independent), AUC-PR (better for imbalance), Brier score. For ranking, NDCG / MAP. For decisions, expected cost / value.

5. Q. "AUC-ROC vs AUC-PR — when each?"
   Show answer

   AUC-ROC is invariant to class balance. AUC-PR (precision-recall curve area) is more sensitive to positive-class performance under heavy imbalance — use it when positives are rare and you care about them (rare-event detection, fraud, ad click).

6. Q. "Your test set AUC is great but the model fails in production. Why?"
   Show answer

   Most common: distribution shift. Train/test were i.i.d. from logged data; production has different traffic mix (covariate shift), different label distribution (prior shift), or selection bias (you only have labels on items the old model showed — exposure bias). Other: temporal leakage (future info in features), train/serve skew (different feature pipelines).

7. Q. "What is the bias of an estimator?"
   Show answer

   The expected difference between the estimator and the true value: Bias = E[θ̂] − θ_true. Distinct from bias-variance decomposition's "bias" (which is squared at a single x). MLE is asymptotically unbiased under regularity conditions, but at finite n can be biased (e.g., MLE of Gaussian variance underestimates).

8. Q. "Why does Adam converge to a worse generalization point than SGD on some vision tasks?"
   Show answer

   Adam's per-coordinate adaptive LR can find sharper minima with worse generalization than SGD. Empirically observed (Wilson 2017, "The Marginal Value of Adaptive Gradient Methods"). LLMs / transformers go the other way (Adam beats SGD); the differences come down to noise structure + curvature shape per problem class. Use AdamW with proper weight decay if Adam.

9. Q. "Why does softmax + cross-entropy have a clean gradient?"
   Show answer

   Let p = softmax(z) and L = −Σ y log p. Then ∂L/∂z = p − y. The complex Jacobian of softmax cancels with the log inside CE. This is why CE is preferred over MSE for classification — gradient signal is simple and proportional to error.

10. Q. "When is Naive Bayes optimal?"
    Show answer

    When features are conditionally independent given the class. In practice, NB often works well even when this assumption is violated, because for classification we only need the argmax of P(class | x) — not the calibrated probability. NB's biased probability estimates can still produce correct argmax.

11. Q. "Why does the bias term in linear regression matter?"
    Show answer

    Without bias (intercept), the regression line is forced through the origin — generally a worse fit. With bias, the model fits an arbitrary affine function. In NNs with biased layers, removing biases ties the layer's "default" output to zero, which is rarely what you want — though some modern transformers (Llama) drop bias terms in linear layers because RMSNorm makes them redundant.

12. Q. "Explain log-sum-exp trick."
    Show answer

    log Σ exp(x_i) overflows if any x_i is large. Subtract max: log Σ exp(x_i) = m + log Σ exp(x_i − m) where m = max(x_i). Now exp arguments are ≤ 0, can't overflow. Used in softmax, partition functions, and any "log of a sum of exponentials" computation.

13. Q. "What's the curse of dimensionality?"
    Show answer

    In high-dim space, (1) volume concentrates on the surface of any region, (2) all pairwise distances become similar (concentration of measure), (3) the number of samples needed to cover the space grows exponentially, (4) most local methods (kNN, kernel density) fail. Why deep learning still works: data lies on a low-dim manifold within the high-dim space.

14. Q. "Why is GBM more sensitive to LR than RF?"
    Show answer

    GBM's trees are sequential — each fits the residual after applying previous trees with shrinkage η. Small η = small steps, more trees needed, less overfit per step. RF trees are independent — no LR; aggregation reduces variance regardless. GBM's η + n_estimators control bias-variance trade.

15. Q. "Explain regularization paths."
    Show answer

    The set of solutions w(λ) as the regularization strength λ varies from 0 to ∞. For lasso, the path is piecewise linear in λ (LARS algorithm). For ridge, smooth. Useful for hyperparam selection (CV over the path) and feature ranking (order in which lasso adds features).

16. Q. "Why does the kernel trick let SVMs handle non-linear data?"
    Show answer

    Map data implicitly to a high-dim feature space φ(x); a linear separator in φ-space corresponds to a non-linear boundary in original space. The trick avoids explicit φ — only inner products are needed, and K(x,x') = ⟨φ(x), φ(x')⟩ is computed directly. RBF kernel corresponds to infinite-dim feature space.

17. Q. "Difference between generative and discriminative models?"
    Show answer

    Generative models P(x, y) (or P(x|y) and P(y)). Examples: Naive Bayes, GMMs, HMMs, VAEs, diffusion. Can sample x. Discriminative models P(y | x) directly. Examples: logistic regression, SVM, neural classifiers. Generally better classification accuracy with enough data; generative wins with little data or when you need to generate. (Ng & Jordan 2002.)

18. Q. "What does it mean for an estimator to be 'consistent'?"
    Show answer

    It converges in probability to the true parameter as n → ∞: P(|θ̂_n − θ| > ε) → 0. Distinct from unbiasedness — biased estimators can be consistent (bias goes to 0). MLE is consistent under regularity conditions; biased shrinkage estimators (like ridge) are inconsistent for the true coefficient but can have lower MSE.

19. Q. "What is the bias of a sample variance? How do you fix it?"
    Show answer

    Sample variance with 1/n divisor is biased: E[s²] = (n−1)/n · σ². Bessel's correction divides by (n−1) instead of n, yielding unbiased s² = (1/(n−1)) Σ (x_i − x̄)². Intuition: x̄ is computed from the sample, so deviations from x̄ are slightly smaller than from true μ.

20. Q. "Why don't we use accuracy as a loss function?"
    Show answer

    Accuracy is non-differentiable (involves argmax). Gradient zero almost everywhere. Need a smooth surrogate (cross-entropy, hinge, softmax) that's differentiable and shares argmax with accuracy. Surrogate losses' relationship to the target metric is studied as "calibration" / "consistency" in the surrogate-loss literature.

21. Q. "When is a problem 'convex'? Why do we care?"
    Show answer

    The objective is a convex function and the feasible set is convex. Implications: any local minimum is global; gradient descent converges to optimum; saddle points don't exist (in strict convex case); duality gap is zero. Linear/logistic regression with L2 is convex. Deep NNs are non-convex — much of optimization theory doesn't apply directly, hence the focus on empirical heuristics.

22. Q. "What does it mean for two random variables to be 'uncorrelated'? Same as 'independent'?"
    Show answer

    Uncorrelated: Cov(X, Y) = 0 ⟺ E[XY] = E[X] E[Y]. Independent: P(X, Y) = P(X) P(Y) (full distribution factors). Independence implies uncorrelated; the converse holds only for jointly Gaussian X, Y. Classic counterexample: X ~ Uniform(−1, 1), Y = X². Cov(X, Y) = 0 but X and Y are perfectly dependent.

23. Q. "What is the variance of a sum of correlated random variables?"
    Show answer

    Var(Σ X_i) = Σ Var(X_i) + 2 Σ_{i<j} Cov(X_i, X_j). For uncorrelated, the cross-covariance terms vanish. This is why ensembling (random forest) reduces variance more when base learners are decorrelated.

24. Q. "What's the difference between L1 and L2 from a gradient perspective?"
    Show answer

    L2 gradient: 2λθ — proportional to weight; pulls weights uniformly toward zero, never reaching it. L1 (sub)gradient: λ · sign(θ) — constant pull regardless of magnitude; small weights get pushed to exactly 0 by the soft-threshold operator sign(θ) max(|θ| − λ, 0). That's why L1 induces sparsity.

25. Q. "What's the role of the Hessian in second-order optimization?"
    Show answer

    Newton's update: θ ← θ − H⁻¹ ∇L. H⁻¹ rescales gradient by the inverse local curvature → quadratic convergence near optimum. Issue: H is N×N (infeasible for N≫1000). Approximations: BFGS (rank-1 updates from gradient history), Adam (diagonal preconditioner using grad²), Shampoo (Kronecker-factored), K-FAC (block-diagonal Fisher).

26. Q. "Explain Jensen's inequality and one ML application."
    Show answer

    For convex f: f(E[X]) ≤ E[f(X)]. Concave: reverse. ML application — ELBO derivation: log P(x) = log ∫ P(x, z) dz = log ∫ q(z) P(x,z)/q(z) dz ≥ ∫ q(z) log P(x,z)/q(z) dz (by Jensen on log, which is concave). The RHS is the ELBO. This is how the VAE's variational bound is derived.

27. Q. "Why is cross-entropy non-negative?"
    Show answer

    For probability distributions p (true) and q (predicted): H(p, q) ≥ H(p) ≥ 0. The first inequality is Gibbs' inequality (equivalent to KL ≥ 0). The second is because log p ≤ 0 for p ∈ [0,1]. Cross-entropy = 0 only when p is deterministic AND q matches.

28. Q. "Why is KL divergence asymmetric?"
    Show answer

    KL(p||q) = Σ p log(p/q) uses p as the weighting. KL(q||p) uses q. They penalize different regions: KL(p||q) heavily penalizes q being small where p is large (covers all modes of p — "mode-covering"); KL(q||p) heavily penalizes q being large where p is small (q sticks to one mode — "mode-seeking"). Choice matters in VI: ELBO uses KL(q||p) → mode-seeking; EP uses KL(p||q) → mode-covering.

29. Q. "What is the difference between batch norm at train vs eval?"
    Show answer

    Train: normalize using batch statistics (mean/var of current mini-batch). Eval: normalize using running averages accumulated during training. Differences cause subtle bugs: a model in train mode at inference will use one-batch stats and behave differently. Always call model.eval() for inference.

30. Q. "Why doesn't increasing model capacity always help in deep learning?"
    Show answer

    Classical view: bigger capacity → overfit. But for over-parameterized NNs, double descent shows test error rises near interpolation threshold, then falls. Practical issues: (1) data bottleneck — bigger model with same data doesn't help past a point (Chinchilla scaling); (2) optimization gets harder; (3) inference cost dominates total cost. The right framing is "compute optimal" — fix budget, find best model size + data size jointly.

31. Q. "What's the cost of training a logistic regression vs an SVM at scale?"
    Show answer

    Logistic regression: convex, smooth — SGD scales linearly with N samples. SVM: convex, non-smooth (hinge) — solved as QP via SMO or via SGD on the hinge loss. Linear SVM and logistic at scale are essentially equivalent in cost; kernel SVM is O(N²) memory for the Gram matrix → infeasible past ~100k samples. Modern recsys/CTR pipelines use logistic regression (or its NN generalization) for this reason.

32. Q. "Why are random forests said to be 'almost impossible to overfit'?"
    Show answer

    Adding more trees doesn't increase model variance — bagging averages out variance. But each tree can overfit individually if grown unrestricted; the variance of the ensemble is tamed by averaging. RF still can overfit if (a) too few trees, (b) trees too deep relative to data, (c) too many irrelevant features at each split. "Hard to overfit" is a slight overstatement.

33. Q. "What's the relationship between sigmoid and softmax?"
    Show answer

    Softmax with 2 classes is equivalent to sigmoid: softmax([z, 0]) = [σ(z), 1−σ(z)]. Sigmoid is the binary special case. For binary classification, sigmoid is more efficient (one logit instead of two with the constraint they sum to 1).

34. Q. "Why does feature engineering still matter when we have deep learning?"
    Show answer

    For tabular data with weak signal-to-noise, feature engineering (interactions, log transforms, target encoding, time-aggregations) outperforms NNs at modest data sizes. NNs need lots of data to learn what good features are; trees + good features beat NNs + raw data on most tabular tasks. Even at scale, domain-specific feature engineering speeds learning and improves robustness.

35. Q. "What's the difference between standardization, normalization, and whitening?"
    Show answer

    Standardization: zero mean, unit variance per feature. Min-max normalization: rescale to [0,1] (or [−1,1]). Whitening: full covariance is identity (decorrelates features). PCA whitening uses eigenvectors; ZCA whitening preserves the original feature space orientation. NN inputs typically use standardization; whitening is rare for NNs because BN partly accomplishes it inside the network.

36. Q. "What's wrong with using R² for non-linear models?"
    Show answer

    R² = 1 − SS_res/SS_tot. For linear regression with intercept on training data, R² ∈ [0, 1] and equals squared Pearson correlation between y and ŷ. For non-linear / no-intercept / out-of-sample, R² can be negative (model worse than mean predictor). Better to report MSE/MAE plus context.

37. Q. "What does 'data leakage' mean? Common sources?"
    Show answer

    Train data contains information about the label or test set that wouldn't be available at prediction time. Sources: (1) target included in features (oops); (2) data preprocessing fit on full dataset before split (e.g., scaling, target encoding); (3) future info in time-series features; (4) duplicates between train and test; (5) selection bias (only labeled examples are easy ones); (6) for recsys: training on logs that the model itself created.

38. Q. "Explain stratified vs random K-fold cross-validation."
    Show answer

    Stratified: each fold has the same class distribution as the full dataset. Critical for imbalanced classification — random folds can leave a fold with 0 positive examples. For regression, can stratify on binned target. For time-series, neither — use time-series CV (forward-chaining splits).

39. Q. "Why might precision and recall not be enough?"
    Show answer

    P/R fix a single threshold. AUC summarizes across thresholds. But neither captures: (1) cost asymmetry (false positive cost ≠ false negative cost — use expected cost), (2) ranking quality beyond binary (NDCG), (3) calibration of probabilities, (4) per-segment performance (slicing matters). For recommendation, P@K is misleading without considering position bias.

40. Q. "What is a 'sufficient statistic'?"
    Show answer

    A function T(X) of the data such that the conditional distribution of X given T(X) doesn't depend on the parameter θ. All info about θ in the data is captured by T(X). For Gaussian with known variance, sample mean is sufficient for μ. Comes up in exponential family / GLM theory.

41. Q. "What's the connection between linear regression and the projection matrix?"
    Show answer

    OLS solution β̂ = (XᵀX)⁻¹ Xᵀy. Predictions ŷ = X β̂ = X (XᵀX)⁻¹ Xᵀ y = H y where H = X(XᵀX)⁻¹Xᵀ is the "hat" matrix — the projection onto the column space of X. So OLS projects y onto the column space of X. I − H projects onto the residual space. Sum of leverages = trace(H) = rank(X) = effective parameter count.

42. Q. "What's the multiple-comparisons problem in A/B testing?"
    Show answer

    Running many tests at α = 0.05, you'll get false positives at rate ~5% per test → with 20 tests, ~63% chance of at least one false positive. Corrections: Bonferroni (divide α by # tests, conservative), Benjamini-Hochberg (controls false discovery rate, less conservative), sequential testing methods. Always pre-register your primary metric and a small set of guardrails.

43. Q. "What does it mean for a model to be 'identifiable'?"
    Show answer

    Different parameter values yield different distributions: P(x | θ_1) ≠ P(x | θ_2) implies θ_1 ≠ θ_2. Without identifiability, you can't uniquely recover θ from data even with infinite samples. Mixture models are non-identifiable up to label permutation. Deep nets are non-identifiable in the strict sense (many weight configs produce same function); in practice we don't care about θ, only the function.

44. Q. "Why does adding a hidden layer to a neural net make it 'universal'?"
    Show answer

    Universal approximation theorem (Cybenko 1989, Hornik 1991): a feedforward NN with one hidden layer of arbitrary width and a non-polynomial activation can approximate any continuous function on a compact set to arbitrary accuracy. Theoretical result; practically the required width is often exponential. Why depth helps: depth gives exponentially more efficient representation than width for many functions (Telgarsky 2016).

45. Q. "What's the empirical risk vs the true risk?"
    Show answer

    True risk: R(f) = E_{(x,y)~P}[L(f(x), y)]. What you actually want — expected loss on the data distribution. Empirical risk: R̂(f) = (1/n) Σ L(f(x_i), y_i). What you compute on the training set. ERM minimizes R̂; learning theory studies how close R̂ is to R (generalization gap).

46. Q. "What's the difference between online learning and online inference?"
    Show answer

    Online learning: model updates from each example as it arrives (streaming SGD). Online inference: model serves predictions in real time (latency budget, no retraining). They're orthogonal — most production systems do online inference + batch retraining; full online learning is rare due to stability concerns.

47. Q. "What is the curse of binarization for ranking metrics?"
    Show answer

    Binary metrics (precision@K, recall@K) treat all relevant items as equal. NDCG / graded relevance accounts for varying degrees of relevance. For ranking, the difference between rank-1 and rank-2 of a great item matters more than rank-99 vs rank-100 — discount functions (1/log₂(rank+1) for NDCG) capture this; binary metrics don't.

48. Q. "Why doesn't increasing regularization always reduce overfitting?"
    Show answer

    Past a point, regularization underfits — increases bias more than it reduces variance. The bias-variance trade-off has a sweet spot; CV finds it. Also, the form of regularization matters — L2 may not produce the right inductive bias for sparse / structured data, where L1 / group-lasso would.

Extended quiz — 30+ more theory questions

Cover all of these and you've passed any "ML theory probe" round.

1. Q. "What is overfitting?"
   Show answer

   Model fits training data better than the underlying distribution → poor generalization. Symptoms: large train-test gap. Causes: high model capacity, low data, noisy labels, leakage. Cures: regularization, more data, augmentation, early stopping, ensembling, simpler model.

2. Q. "Why does dropout reduce overfitting?"
   Show answer

   Stochastic ensemble interpretation (each mask = sub-network); co-adaptation prevention; Bayesian variational interpretation (Gal & Ghahramani 2016). Equivalent to a noise injection regularizer.

3. Q. "What is the difference between parameters and hyperparameters?"
   Show answer

   Parameters: learned from data via optimization (weights, biases). Hyperparameters: set before learning, control the learning process (LR, batch size, regularization strength, architecture). Hyperparameter optimization: grid search, random search, Bayesian optimization (Optuna, Hyperopt), Hyperband, BOHB.

4. Q. "What is gradient clipping and when do you use it?"
   Show answer

   Cap gradient norm at a threshold C: if ||g|| > C, scale g by C/||g||. Used to prevent exploding gradients in RNNs, transformers (rare with proper init + LR), RL. Modern LLM pretraining uses gradient clipping at 1.0 routinely; it's a cheap insurance against loss spikes.

5. Q. "What's a vanishing/exploding gradient and what causes them?"
   Show answer

   Backprop through L layers multiplies L Jacobians. If their product norm is << 1 → vanishing (early-layer weights don't update). If >> 1 → exploding (NaN losses). Causes: bad init (too small / too large), sigmoid/tanh saturation, no normalization, no residual connections. Fixes: He/Xavier init, ReLU/GELU, BN/LN, residuals, gradient clipping.

6. Q. "What's a 'representation' in deep learning?"
   Show answer

   An intermediate layer's activation, viewed as a feature vector for the input. Good representations: linearly separable for downstream tasks; capture invariances; transferable. Self-supervised pretraining (CLIP, SimCLR, MAE, masked LM) explicitly trains for good representations without labels.

7. Q. "What is contrastive learning?"
   Show answer

   Learn representations by pulling similar (positive) pairs together and pushing dissimilar (negative) pairs apart in embedding space. InfoNCE loss is the dominant form. Examples: SimCLR (image augmentations as positives), CLIP (image-caption pairs), two-tower retrieval (user-item interactions).

8. Q. "Why is temperature in softmax / contrastive learning important?"
   Show answer

   Temperature τ scales logits before softmax. Low τ → sharper distribution (focuses on top items). High τ → flatter. In contrastive learning, low τ amplifies the gradient on hard negatives but can over-fit; high τ underweights hard examples. CLIP uses learned τ.

9. Q. "What's the difference between SGD and mini-batch SGD?"
   Show answer

   SGD: gradient on one example. Mini-batch SGD: gradient on a small batch (32-1024 typical). Mini-batch is the standard — vectorized compute, less noisy, can use BN. "SGD" in modern DL almost always means mini-batch SGD.

10. Q. "What is gradient checkpointing?"
    Show answer

    Trade compute for memory: save only some activations during forward; recompute the rest during backward. ~33% extra forward FLOPs for full block checkpointing; ~10× memory savings. Critical for training large models on limited GPU memory.

11. Q. "Explain teacher forcing vs free running in seq2seq training."
    Show answer

    Teacher forcing: at each decode step, feed the ground-truth previous token (not the model's own output). Stable, fast training. Issue: exposure bias — at inference the model sees its own (potentially wrong) outputs. Free running: feed model's own output. Slower training, can be unstable. Modern fixes: scheduled sampling, distillation, RL fine-tuning.

12. Q. "What is label smoothing and when does it help?"
    Show answer

    Replace one-hot target with (1 − ε) · onehot + ε / K. The model is no longer rewarded for assigning probability 1 to the correct class → less overconfident, better calibration. Used in original Transformer, Inception nets. Less common in LLM pretraining where calibration isn't a target metric.

13. Q. "What is mixup and why might it work?"
    Show answer

    Train on convex combinations: x̃ = λ x_i + (1 − λ) x_j, ỹ = λ y_i + (1 − λ) y_j. Encourages linear behavior between training points. Improves generalization, calibration, robustness. Vision-only really; doesn't work for discrete tokens.

14. Q. "What is the manifold hypothesis?"
    Show answer

    Real-world high-dim data (images, text) lies on a low-dimensional manifold within the ambient high-dim space. Implication: ML can succeed despite curse of dimensionality. Most data points are unreachable (gibberish images, random text); the meaningful subset has low intrinsic dimension. Generative models (diffusion, VAE) implicitly learn this manifold.

15. Q. "Why does the LR schedule matter?"
    Show answer

    Early training: high LR for fast progress, but instability. Late training: low LR for fine convergence. Warmup smooths early instability (large grad magnitudes when Adam moments are noisy). Decay (cosine, WSD, linear) reduces LR over time → finer steps near optimum. Empirical: schedule choice affects final loss by 5-15% with the same total compute.

16. Q. "How do you handle class imbalance?"
    Show answer

    (1) Resampling: oversample minority (SMOTE), undersample majority. (2) Class weights in loss. (3) Focal loss (down-weights easy examples). (4) Threshold tuning at inference. (5) Get more minority data. (6) Different loss / metric (PR-AUC instead of accuracy). For very rare events, anomaly detection framing.

17. Q. "What is the bias of cross-validation as an estimator of test error?"
    Show answer

    K-fold CV with K folds uses (1 − 1/K) of data per fold → trained on slightly less data than the full model. Slightly pessimistic bias. LOOCV (K=N) is nearly unbiased but high-variance. K=5 or 10 is the standard compromise.

18. Q. "Why might LASSO hurt prediction accuracy compared to ridge?"
    Show answer

    LASSO's sparsity comes at a cost: it picks one feature among correlated ones (somewhat arbitrarily). If both correlated features are predictive, ridge keeps both (small but non-zero coefficients) → better prediction. Elastic net combines both: shrinkage AND selection.

19. Q. "Explain the concept of 'leakage' in time series."
    Show answer

    Future information appears in features available at prediction time. Examples: rolling statistics computed across the whole dataset (uses future); target aggregates by entity that include the current target; preprocessing fit on full data. Time-series CV (forward-chaining, expanding window) prevents this; "as-of" feature stores enforce point-in-time correctness.

20. Q. "What's the difference between ROC and PR curves?"
    Show answer

    ROC: TPR (recall) vs FPR. Invariant to class balance. PR: precision vs recall. Sensitive to class balance — for rare positives, AUC-PR is more informative than AUC-ROC. PR baseline for random classifier = positive class rate (not 0.5).

21. Q. "Why do we use stochastic gradient descent instead of full-batch?"
    Show answer

    Three reasons: (1) computational — full batch is expensive at scale; (2) statistical — gradient noise acts as a regularizer (finds flatter minima); (3) memory — can't fit full batch. The "right" batch size is often the largest that doesn't sacrifice generalization, scaled with critical batch size theory (McCandlish 2018).

22. Q. "What's different about training time-series vs i.i.d. data?"
    Show answer

    (1) Sampling order matters — can't shuffle. (2) Train/val/test must be split chronologically. (3) Cross-validation: forward-chaining or expanding window. (4) Stationarity assumption — distribution can shift over time, requires periodic retraining. (5) Lag/lead features need careful construction. (6) Target leakage from rolling stats.

23. Q. "Explain Maximum Mean Discrepancy (MMD)."
    Show answer

    A kernel-based distance between two distributions: MMD²(P, Q) = E_{x,x'~P}[k(x,x')] − 2 E_{x~P, y~Q}[k(x,y)] + E_{y,y'~Q}[k(y,y')]. Used for two-sample testing, generative model evaluation, distribution shift detection. With a characteristic kernel (RBF), MMD = 0 iff P = Q.

24. Q. "Why is the Gaussian distribution so prevalent in ML?"
    Show answer

    (1) Central Limit Theorem: sum of independent random variables tends to Gaussian. (2) Maximum entropy distribution given mean and variance constraints. (3) Closed-form posterior in many conjugate models. (4) MLE under Gaussian noise = MSE, the most-used loss. (5) Computationally tractable (multivariate Gaussian closed under marginal/conditional/linear transform).

25. Q. "Why does Bayes' rule matter?"
    Show answer

    P(θ | D) = P(D | θ) P(θ) / P(D). Posterior ∝ likelihood × prior. Foundation of Bayesian ML, MAP estimation, naive Bayes classifier, every probabilistic interpretation in deep learning. The denominator P(D) is the partition function; intractable in general → motivates MCMC, VI, ABC.

Reading: papers and posts that go deep on each topic

• Bias-variance: ESL Ch 7
• L1 vs L2 sparsity: Friedman, Hastie, Tibshirani 2010 — glmnet paper
• Why ReLU: Glorot, Bordes, Bengio 2011 "Deep Sparse Rectifier Neural Networks"
• Generalization theory survey: Kawaguchi, Kaelbling, Bengio 2017
• Why does SGD generalize: Keskar 2017 — Large-batch training; Belkin 2019 — double descent
• BatchNorm "covariate shift" rebutted: Santurkar 2018
• Calibration of NNs: Guo 2017
• Bayesian dropout: Gal & Ghahramani 2016
• Adam vs SGD generalization: Wilson 2017
• Implicit bias of gradient descent: Soudry 2018
• NTK: Jacot, Gabriel, Hongler 2018
• μP: Yang 2022 — Tensor Programs V
• Karpathy's A Recipe for Training Neural Networks — pragmatic ML wisdom
• Lilian Weng's blog — every post is a deep dive: lilianweng.github.io