SUMMARY NOTES · ML THEORY · CS229 ORDER + ESL DEPTH
ML theory — summary notes
Dense, scannable crib-sheets that follow the CS229 lecture order with ESL depth — built to trigger recall, not to be a long course. Each page packs one topic: definitions, the few formulas that matter, the ambiguities cleared up, the interview probe, and a one-line "remember." Logical progression — every page builds on the last.
The progression (why the order matters)
Everything in supervised ML is (model class) + (loss) + (optimizer) + (regularizer). Part 0 builds the lens: decision theory says what we're minimizing, the frequentist/Bayesian split says how we reason about parameters, MLE/MAP says where losses and regularizers come from, and bias–variance says why we can't just minimize training error. Parts I+ then apply that lens to each algorithm in CS229 order.
Part 0 · live
Foundations: probability & stats primer, estimators, decision theory, frequentist vs Bayesian, MLE/MAP, bias–variancePart I · next
Supervised: linear & logistic regression, GLMs, GDA/Naive Bayes, kernels & SVMPart II · next
Optimization: convexity, GD/SGD/Adam, Newton, Lagrange duality & KKTPart III · next
Learning theory: ERM, uniform convergence, VC dimension, PAC; regularization & model selectionPart IV · next
Unsupervised: k-means, EM/ELBO/GMM, PCA, ICAPart V · next
Trees & ensembles: CART, bagging, random forests, AdaBoost, gradient boostingPart VI · next
Reinforcement learning: MDPs, value/policy iteration, Q-learning, policy gradientPart VII · next
Practice: evaluation & calibration, and the "do-you-really-get-it" probesDeep learning gets its own summary-notes page next (backprop, init/normalization, transformers, training).
01
🎯Four knobs run all of supervised ML — model, loss, optimizer, regularizer. Everything else is detail.
A summary note is a memory trigger: if you already learned it, one page snaps it back; if you half-know it, the "clears up" box fixes the fuzzy part. Read top-to-bottom once; thereafter jump to any page.
Every page has the same parts
Setupthe objects (data, model, loss) in 2–3 lines.
Formulasonly the ones you must be able to write.
⚠ Clears upthe exact ambiguity people get wrong.
◆ Probehow an interviewer tests real understanding.
✓ Rememberthe one sentence to carry out of the room.
The one frame for all of supervised ML
Pick a model class $f_\theta$, a loss $L$, an optimizer to minimize it, and a regularizer to control complexity. Almost every method is a choice of these four. The foundations explain where each comes from:
loss= negative log-likelihood of a noise model (ch4).
regularizer= negative log-prior on parameters (ch4).
why not 0 loss= bias–variance & generalization (ch5).
Remember Method = model class + loss + optimizer + regularizer. The first four chapters tell you where the loss and the regularizer secretly come from.
02
🎯Means add always; variances add only when independent — and the CLT makes the average Gaussian with error σ/√n.
The handful of probability facts every derivation leans on. For MLE, the Gaussian noise model, and bias–variance to feel obvious, these have to be reflexes first.
Describing a random variable
PMF / PDF$p(x)$ (discrete) or $f(x)$ (continuous); a density integrates to 1.
CDF$F(x)=P(X\le x)$, increases from 0 to 1.
Expectation$\mathbb{E}[X]=\sum_x x\,p(x)$ or $\int x f(x)\,dx$ — the long-run average.
LOTUS$\mathbb{E}[g(X)]=\sum_x g(x)p(x)$ — no need to find the law of $g(X)$.
Spread & co-movement
$$\operatorname{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2,\qquad \operatorname{Var}(aX{+}b)=a^2\operatorname{Var}(X).$$
$$\operatorname{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y],\qquad \rho=\tfrac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}\in[-1,1].$$
$$\operatorname{Var}(X{+}Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y).$$Independent ⇒ covariance 0 ⇒ variances simply add.
Conditioning, Bayes & the total laws
$$P(A\mid B)=\frac{P(A,B)}{P(B)},\qquad \underbrace{P(A\mid B)}_{\text{posterior}}=\frac{P(B\mid A)\,P(A)}{P(B)},\qquad p(x)=\sum_y p(x,y).$$Bayes' rule and marginalization — the entire Bayesian engine in one line.
independence$p(x,y)=p(x)\,p(y)$: knowing one says nothing about the other.
total expectation$\mathbb{E}[X]=\mathbb{E}\big[\mathbb{E}[X\mid Y]\big]$ — average the conditional averages.
total variance$\operatorname{Var}(X)=\mathbb{E}[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(\mathbb{E}[X\mid Y])$.
The two theorems that make statistics work
LLNthe sample mean $\bar X_n\to\mathbb{E}[X]$ — averages converge to the truth.
CLT$\bar X_n\approx\mathcal{N}(\mu,\,\sigma^2/n)$ for large $n$, whatever $X$'s shape; standard error $=\sigma/\sqrt{n}$.
⚠ Clears up — means vs variances under addition
Expectation is always linear: $\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]$ even for dependent variables. Variance is not — it adds only when the covariance is zero. "Average of correlated things" is easy; "variance of a sum of correlated things" needs the cross term.
⚠ Clears up — independent vs uncorrelated
Independent $\Rightarrow$ uncorrelated, but not the reverse: correlation only sees linear association, so variables can be uncorrelated yet dependent. The one exception: for jointly Gaussian variables, uncorrelated does imply independent.
◆ Interview probe
"$\operatorname{Var}(X+Y)$?" → add $2\operatorname{Cov}(X,Y)$ unless independent. "What does the CLT buy you?" → error bars: the mean of $n$ samples has spread $\sigma/\sqrt{n}$, so 4× the data halves the error.
Remember $\mathbb{E}$ is always linear; $\operatorname{Var}(aX{+}b)=a^2\operatorname{Var}(X)$; independent ⇒ variances add; CLT ⇒ standard error $=\sigma/\sqrt{n}$.
Tricky interview questions 12
$\operatorname{Var}(X+Y)=?$
$\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)$. The cross term only vanishes when $X$ and $Y$ are uncorrelated. Trap: dropping the $2\operatorname{Cov}$ term and writing $\operatorname{Var}(X)+\operatorname{Var}(Y)$ without checking independence.
Is linearity of expectation $\mathbb{E}[X+Y]=\mathbb{E}[X]+\mathbb{E}[Y]$ true for dependent variables?
Yes, always — expectation is linear with no independence needed. Variance is the one that needs independence. Trap: thinking linearity requires independence; it never does.
Does zero correlation imply independence?
No. Correlation only sees linear association. Classic counterexample: $Y=X^2$ with $X$ symmetric about 0 has $\operatorname{Cov}(X,Y)=0$ yet $Y$ is fully determined by $X$. Independence $\Rightarrow$ uncorrelated, not the reverse. Trap: assuming the equivalence holds generally — it holds only in special cases such as jointly Gaussian variables.
Covariance vs correlation — what's the difference?
Covariance measures joint linear variation in raw units (unbounded, scale-dependent); correlation is covariance normalized by $\sigma_X\sigma_Y$ into $[-1,1]$ and is scale-invariant. Trap: comparing covariances across datasets — only correlation is unit-free and comparable.
State the Central Limit Theorem. Why does it matter?
For i.i.d. variables with finite variance, the sample mean $\bar X_n$ is approximately $\mathcal{N}(\mu,\sigma^2/n)$ as $n$ grows, whatever the population's shape. It justifies normal-based confidence intervals and z/t-tests on non-normal data. Trap: claiming the CLT makes the raw data normal — it's about the distribution of the mean. Also fails for infinite variance (Cauchy).
What does the CLT actually buy you in one number?
Error bars. The mean of $n$ samples has standard error $\sigma/\sqrt{n}$, so $4\times$ the data halves the error. Trap: thinking more data shrinks error linearly — it shrinks like $1/\sqrt{n}$, so diminishing returns.
Law of Large Numbers vs CLT?
LLN says $\bar X_n\to\mu$ (convergence to a point). CLT goes further: the leftover wobble around $\mu$ is approximately $\mathcal{N}(\mu,\sigma^2/n)$ — it gives the shape and scale, not just the limit. Trap: gambler's fallacy — LLN is about long-run averages, not short runs "evening out."
State Bayes' theorem and solve: 0.1% prevalence, 99% sensitivity, 99% specificity — P(disease | positive)?
$P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}$. Here $\frac{0.99\cdot0.001}{0.99\cdot0.001+0.01\cdot0.999}\approx 9\%$ — false positives from the huge healthy population swamp the few true positives. Trap: base-rate fallacy — answering $\sim99\%$ by ignoring the tiny prior.
What is LOTUS and why is it useful?
Law Of The Unconscious Statistician: $\mathbb{E}[g(X)]=\sum_x g(x)\,p(x)$ — you compute the expectation of $g(X)$ from $X$'s own law, without ever finding the distribution of $g(X)$. Trap: assuming $\mathbb{E}[g(X)]=g(\mathbb{E}[X])$ — false unless $g$ is linear (Jensen's inequality).
What is the law of total variance?
$\operatorname{Var}(X)=\mathbb{E}[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(\mathbb{E}[X\mid Y])$ — total spread = average within-group spread + spread of the group means. Total expectation is the simpler twin: $\mathbb{E}[X]=\mathbb{E}[\mathbb{E}[X\mid Y]]$. Trap: keeping only the first term and forgetting the between-group variance of the conditional means.
Probability vs likelihood — what's the distinction?
In probability, $\theta$ is fixed and you ask how probable the data is: $P(\text{data}\mid\theta)$. Likelihood fixes the observed data and reads the same expression as a function of $\theta$: $L(\theta)=P(\text{data}\mid\theta)$. Trap: treating likelihood as a distribution over $\theta$ — it does not integrate to 1 over $\theta$.
Roll a fair die three times. P(at least one six)? P(two sixes in a row)?
At least one six via the complement: $1-(5/6)^3=91/216$. Two-in-a-row over three rolls: patterns $66X$ and $X66$ give $6+6=12$, minus the double-counted $666$, so $11/216$. Trap: computing "at least one" the hard way instead of $1-P(\text{none})$, and forgetting to subtract the double-counted $666$.
03
🎯The Gaussian is the default because sums of almost anything drift Gaussian (CLT).
You don't need every distribution — just the dozen that recur as noise models, labels, and priors. Know each one's mean, variance, and the ML slot it fills.
The ones that actually show up
| Distribution | Mean / Var | Shows up as |
|---|---|---|
| Bernoulli($p$) | $p$ / $p(1{-}p)$ | a binary label; the head of logistic regression |
| Binomial($n,p$) | $np$ / $np(1{-}p)$ | count of successes in $n$ trials |
| Categorical / Multinomial | — | multiclass labels; the softmax head |
| Gaussian $\mathcal{N}(\mu,\sigma^2)$ | $\mu$ / $\sigma^2$ | regression noise (→ MSE); the CLT limit |
| Poisson($\lambda$) | $\lambda$ / $\lambda$ | counts & rates; Poisson regression |
| Exponential($\lambda$) | $1/\lambda$ / $1/\lambda^2$ | waiting times; memoryless |
| Beta($\alpha,\beta$) | on $[0,1]$ | conjugate prior for a Bernoulli/Binomial $p$ |
| Gamma | on $[0,\infty)$ | conjugate prior for a rate / precision |
| Laplace | heavy-tailed | the L1 prior; MAE noise model |
Two ideas that tie them together
exponential familyGaussian, Bernoulli, Poisson… share one form $p(y;\eta)=b(y)\,e^{\eta^\top T(y)-a(\eta)}$ — the backbone of GLMs.
conjugacya prior is conjugate if the posterior stays in the same family → closed-form Bayesian updates: Beta–Bernoulli, Gaussian–Gaussian, Gamma–Poisson.
⚠ Clears up — Bernoulli vs Binomial vs Categorical
Bernoulli = one yes/no trial. Binomial = the count of yeses over $n$ independent Bernoulli trials. Categorical = one draw from $K$ classes (the multiclass Bernoulli); Multinomial = counts over $n$ categorical draws.
⚠ Clears up — why the Gaussian is everywhere
Three reasons: the CLT makes sums/averages drift Gaussian; it is the maximum-entropy distribution for a fixed mean and variance (the least-assuming choice); and it is closed under linear operations (sums and linear maps of Gaussians stay Gaussian), which keeps the algebra tractable.
◆ Interview probe
"Mean and variance of a Bernoulli?" → $p$ and $p(1{-}p)$ (max variance at $p=\tfrac12$). "Conjugate prior for a Bernoulli likelihood?" → the Beta — the posterior is Beta with updated counts.
Remember Bernoulli $p(1{-}p)$, Gaussian (CLT/noise), Poisson $\lambda$. Conjugate pairs (Beta–Bernoulli, Gaussian–Gaussian) give closed-form posteriors; the GLM workhorses are all exponential-family.
Tricky interview questions 12
Compare Bernoulli, Binomial, and Poisson. Give each one's mean and variance.
Bernoulli($p$): one yes/no trial, mean $p$, var $p(1{-}p)$. Binomial($n,p$): sum of $n$ i.i.d. Bernoullis, mean $np$, var $np(1{-}p)$. Poisson($\lambda$): counts of rare events in a fixed window, mean $=$ var $=\lambda$. Binomial $\to$ Poisson as $n\to\infty,\ p\to 0$ with $np=\lambda$ fixed. Trap: mixing up which one has mean $=$ variance (that's Poisson), or forgetting Binomial is a sum of i.i.d. Bernoullis.
Bernoulli vs Binomial vs Categorical vs Multinomial — what's the difference?
Bernoulli $=$ one binary trial. Binomial $=$ count of successes over $n$ Bernoulli trials. Categorical $=$ one draw from $K$ classes (the multiclass Bernoulli). Multinomial $=$ counts over $n$ categorical draws. Trap: calling a single $K$-way draw "multinomial" — that's categorical; multinomial needs the $n$ counts.
Why is the Gaussian the default distribution everywhere in ML?
Three reasons. (1) CLT: sums/averages of almost anything drift Gaussian. (2) Max-entropy: it's the least-assuming distribution for a fixed mean and variance. (3) Closed under linear maps: sums and linear transforms of Gaussians stay Gaussian, so the algebra stays tractable. Trap: citing only the CLT and forgetting it's the maximum-entropy choice for a given mean/variance.
What does the Central Limit Theorem actually say, and what does it buy you?
For i.i.d. variables with finite variance, the sample mean $\to\mathcal{N}(\mu,\sigma^2/n)$ as $n$ grows, whatever the population's shape. It buys you error bars: the standard error is $\sigma/\sqrt n$, so 4$\times$ the data halves the error. Trap: saying the CLT makes the raw data normal — it's about the distribution of the mean. It also fails when variance is infinite (e.g. Cauchy).
How does the Law of Large Numbers differ from the CLT?
LLN: the sample mean converges to the true mean $\mu$ (a point). CLT: it characterizes the fluctuations around $\mu$ as approximately $\mathcal{N}(\mu,\sigma^2/n)$ (the shape and scale). LLN says "it gets there"; CLT says "here's how it wiggles on the way." Trap: the gambler's fallacy — LLN is about long-run averages, not short-run outcomes "evening out."
What distribution models waiting times, and what makes it special?
The Exponential($\lambda$): mean $1/\lambda$, variance $1/\lambda^2$. It's memoryless — $P(X>s{+}t\mid X>s)=P(X>t)$, so having waited gives you no credit toward the next event. It's the continuous twin of the geometric and the gap distribution of a Poisson process. Trap: thinking "I've waited a long time, an event is overdue" — memorylessness means the clock resets every instant.
What is a conjugate prior, and why do practitioners love them?
A prior is conjugate to a likelihood if the posterior stays in the same family, giving closed-form Bayesian updates (no integration). Key pairs: Beta–Bernoulli/Binomial, Gaussian–Gaussian, Gamma–Poisson. E.g. Beta($\alpha,\beta$) updates to Beta($\alpha{+}k,\ \beta{+}n{-}k$) after $k$ successes in $n$ trials. Trap: assuming conjugacy is fundamental — it's a mathematical convenience for tractability, not a modeling requirement.
What's the conjugate prior for a Bernoulli likelihood, and what's the posterior?
The Beta($\alpha,\beta$) on $[0,1]$. After observing $k$ successes in $n$ trials, the posterior is Beta($\alpha{+}k,\ \beta{+}n{-}k$) — you just add the success/failure counts. The hyperparameters $\alpha,\beta$ act like pseudo-counts of prior successes and failures. Trap: naming the Gaussian — Gaussian is conjugate to a Gaussian mean, not to a Bernoulli probability.
What is the exponential family, and why does it matter for ML?
A family written as $p(y;\eta)=b(y)\,e^{\eta^\top T(y)-a(\eta)}$ — Gaussian, Bernoulli, Poisson, Gamma, categorical are all members, differing only in $b,T,a$. It's the backbone of GLMs: pick a family, point its natural parameter at $\theta^\top x$, and every model shares the gradient $(y-\hat y)x$. Trap: thinking it's an obscure curiosity — it's exactly what unifies linear, logistic, and Poisson regression.
Derive the MLE for the parameter $p$ of a Bernoulli from $k$ successes in $n$ trials.
Log-likelihood $\ell(p)=k\log p+(n{-}k)\log(1{-}p)$. Differentiate and set to zero: $k/p-(n{-}k)/(1{-}p)=0\Rightarrow\hat p=k/n$, the sample proportion. The second derivative is negative, confirming a max. Trap: skipping the log (products are painful to differentiate) or jumping to $k/n$ without showing the work.
Which distribution is the L1 prior, and which noise model gives MAE?
The Laplace distribution — heavy-tailed, peaked. As a prior on weights it's the $\ell_1$/lasso penalty (sparsity); as a noise model its negative log-likelihood is the absolute error (MAE), whose optimum is the conditional median (robust to outliers). Trap: confusing it with the Gaussian — Gaussian gives $\ell_2$/ridge and MSE (the mean), not L1/MAE.
For a Bernoulli, where is the variance maximized, and why does that matter?
Variance $p(1{-}p)$ is maximized at $p=\tfrac12$ (value $0.25$) and shrinks to 0 at $p=0$ or $1$. It matters for sample-size/A-B-test planning: the hardest, highest-variance case is a 50/50 split, so power calculations use $p=0.5$ as the conservative worst case. Trap: assuming variance grows with $p$ — it's a downward parabola, peaking in the middle, not at the extremes.
04
🎯An estimator is a random variable; MSE = bias² + variance — so a little bias can buy a lot of variance.
An estimator turns data into a guess at a parameter — and because the data is random, the estimator is itself a random variable. Judging estimators (bias, variance, MSE) is the frequentist lens that MLE and MAP live inside.
An estimator is a random variable
$\hat\theta=\hat\theta(X_1,\dots,X_n)$ is a function of the random sample, so it has a sampling distribution. We grade it by how that distribution sits around the true $\theta$:
$$\operatorname{Bias}(\hat\theta)=\mathbb{E}[\hat\theta]-\theta,\qquad \operatorname{MSE}(\hat\theta)=\mathbb{E}[(\hat\theta-\theta)^2]=\operatorname{Bias}(\hat\theta)^2+\operatorname{Var}(\hat\theta).$$Same decomposition as bias–variance — but the random object here is the estimate, not a prediction.
consistent$\hat\theta\to\theta$ as $n\to\infty$ — more data drives it to the truth.
efficientsmallest variance; the Cramér–Rao bound $\operatorname{Var}(\hat\theta)\ge 1/I(\theta)$ ($I$ = Fisher information) is the floor for unbiased estimators.
standard error$\operatorname{SE}=\operatorname{SD}(\hat\theta)$; a CI is $\hat\theta\pm z\cdot\operatorname{SE}$.
MLE — the default, and its guarantees
$\hat\theta_{\text{MLE}}=\arg\max_\theta p(\text{data}\mid\theta)$. Under regularity it is the gold standard asymptotically:
- Consistent — converges to the true $\theta$.
- Asymptotically normal — $\hat\theta\approx\mathcal{N}(\theta,\,I(\theta)^{-1}/n)$.
- Asymptotically efficient — attains the Cramér–Rao bound.
- Invariant — the MLE of $g(\theta)$ is $g(\hat\theta_{\text{MLE}})$.
But in finite samples it can be biased and it can overfit — it trusts the data completely.
MAP & Bayesian point estimates
Add a prior and summarize the posterior $p(\theta\mid\text{data})$ with one number: the posterior mean (minimizes MSE), the MAP (the mode), or the posterior median. As the foundations show, $\text{MAP}=\text{MLE}+\log\text{-prior}$ — a regularized MLE pulled toward the prior.
⚠ Clears up — "unbiased" is not "best"
The headline of this page. Since $\operatorname{MSE}=\operatorname{bias}^2+\operatorname{variance}$, deliberately accepting a little bias can slash variance and give lower MSE than any unbiased estimator. That is exactly what ridge, MAP, and the James–Stein estimator do — shrink toward zero/prior and win on total error.
⚠ Clears up — the MLE can be biased
Classic case: the MLE of a Gaussian's variance divides by $n$ and is biased low; the usual sample variance divides by $n{-}1$ to fix it. MLE optimality is an asymptotic ($n\to\infty$) promise, not a finite-sample one.
◆ Interview probe
"Is the MLE unbiased?" → not in general; it is consistent and asymptotically unbiased. "Why ever prefer a biased estimator?" → lower MSE via the bias–variance trade (ridge/MAP). "Bias of the Gaussian-variance MLE?" → low by a factor $\tfrac{n-1}{n}$.
Remember An estimator is a random variable; $\operatorname{MSE}=\operatorname{bias}^2+\operatorname{variance}$. MLE = consistent + asymptotically efficient (but can be biased); MAP = MLE + prior, trading a little bias for less variance.
Tricky interview questions 11
What's the difference between MLE and MAP?
MLE maximizes the likelihood $P(D\mid\theta)$; MAP maximizes the posterior $P(\theta\mid D)\propto P(D\mid\theta)\,P(\theta)$, i.e. MLE plus a prior. A zero-mean Gaussian prior gives L2 (ridge), a Laplace prior gives L1, and a flat prior makes MAP $=$ MLE. Trap: calling MAP "always better," or confusing MAP (one point — the posterior mode) with full Bayesian inference (the whole posterior).
Probability vs likelihood — what's the distinction?
Same expression $P(\text{data}\mid\theta)$, read two ways. Probability fixes $\theta$ and varies the data; likelihood fixes the observed data and varies $\theta$. Trap: treating the likelihood as a distribution over $\theta$ — it is not, and it does not integrate to 1 over $\theta$.
Is the MLE unbiased?
Not in general. The MLE is consistent and asymptotically unbiased, but in finite samples it can be biased — e.g. the Gaussian-variance MLE divides by $n$ and is low by a factor $\tfrac{n-1}{n}$, which is why we use $n{-}1$. Trap: quoting MLE's optimality (efficient, normal) as a finite-sample guarantee — those are $n\to\infty$ promises.
Why would you ever prefer a biased estimator?
Because $\operatorname{MSE}=\operatorname{bias}^2+\operatorname{variance}$, so accepting a little bias can slash variance and give lower total error. Ridge, MAP, and James–Stein all shrink toward zero/prior and beat the unbiased estimator on MSE. Trap: equating "unbiased" with "best" — unbiasedness alone says nothing about variance or MSE.
Write the bias–variance decomposition of an estimator's MSE.
$\operatorname{MSE}(\hat\theta)=\mathbb{E}[(\hat\theta-\theta)^2]=\operatorname{Bias}(\hat\theta)^2+\operatorname{Var}(\hat\theta)$, where $\operatorname{Bias}=\mathbb{E}[\hat\theta]-\theta$. Same algebra as the prediction bias–variance split, but here the random object is the estimate, not a prediction. Trap: forgetting the estimator is a random variable (it depends on the random sample) — that randomness is the whole reason it has a variance.
Derive the MLE for the Bernoulli parameter $p$.
With $k$ successes in $n$ trials, $\ell(p)=k\log p+(n{-}k)\log(1{-}p)$; set $\ell'(p)=\tfrac{k}{p}-\tfrac{n-k}{1-p}=0\Rightarrow \hat p=\tfrac{k}{n}$, the sample proportion (and $\ell''<0$ confirms a max). Trap: skipping the log (the product is painful to differentiate), or jumping to $k/n$ without showing it's a maximum.
What does it mean for an estimator to be consistent and efficient?
Consistent: $\hat\theta\to\theta$ as $n\to\infty$ (more data drives it to the truth). Efficient: it attains the smallest possible variance — the Cramér–Rao bound $\operatorname{Var}(\hat\theta)\ge 1/I(\theta)$, where $I$ is the Fisher information. Trap: conflating the two — a consistent estimator can still be inefficient (larger variance than the floor), and the CR bound is the floor for unbiased estimators.
How does MAP connect to regularization?
$\hat\theta_{\text{MAP}}=\arg\min_\theta\big[\sum_i -\log p(y_i\mid x_i,\theta)\;-\;\log p(\theta)\big]$ — the loss is $-\log$ likelihood and the penalty is $-\log$ prior. Gaussian prior $\to$ L2 with $\lambda=\sigma^2/\tau^2$, Laplace prior $\to$ L1. Trap: not seeing that a tighter prior (smaller $\tau$) means a larger $\lambda$ — stronger belief that weights are near zero is literally more shrinkage.
Frequentist vs Bayesian — what's random in each?
Frequentist: the data is random, $\theta$ is a fixed unknown → MLE, confidence intervals. Bayesian: $\theta$ is random (you have a prior), the data is fixed → posterior, MAP/posterior-mean, credible intervals. Trap: saying a 95% credible interval and a 95% confidence interval mean the same thing — only the credible interval supports "95% probability $\theta$ is in here."
What's the CLT, and how does it relate to estimators?
For i.i.d. data with finite variance, the sample mean $\to\mathcal{N}(\mu,\sigma^2/n)$ as $n$ grows — it gives the sampling distribution of an estimator, so the standard error is $\sigma/\sqrt{n}$ and a CI is $\hat\theta\pm z\cdot\operatorname{SE}$. Trap: claiming the CLT makes the raw data normal (it's about the mean's distribution), or that $n=30$ always works — it fails for undefined variance (Cauchy).
Does zero correlation imply independence?
No. Independence implies zero correlation, but not the reverse — correlation only sees linear association, so $Y=X^2$ with $X$ symmetric about 0 is uncorrelated yet fully dependent. The one exception: jointly Gaussian variables, where uncorrelated does imply independent. Trap: asserting uncorrelated means independent in general — it's true only in special cases like joint Gaussianity.
05
🎯Your loss picks the target: squared → mean, absolute → median, 0/1 → mode.
Before any algorithm: data come from an unknown distribution, and we want a predictor that is good on average over future data. That single idea defines the target every method is approximating. (CS229 supervised-learning frame · ESL §2.4.)
Setup
Pairs $(x,y)\sim \mathcal{D}$, an unknown joint distribution. A predictor $f$ pays a loss $L(y,f(x))$. Its quality is the risk = expected loss:
$$ R(f) \;=\; \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[\,L(y,f(x))\,\big]. $$We never know $\mathcal{D}$, so we can't compute $R$ — we minimize an estimate of it.
The empirical risk on a training set of $n$ points is the average loss we can compute:
$$ \hat R(f)=\tfrac1n\textstyle\sum_{i=1}^n L\big(y_i,f(x_i)\big). $$ERM = pick $f$ minimizing $\hat R$. Generalization = how close $\hat R$ stays to $R$ (ch5, learning theory).
The best possible predictor (Bayes) depends only on the loss
Minimizing risk pointwise gives the Bayes predictor $f^\*(x)$ — and which summary of $p(y\mid x)$ it is depends entirely on $L$:
| Loss $L(y,a)$ | Bayes-optimal $f^\*(x)$ | Irreducible error |
|---|---|---|
| squared $(y-a)^2$ | the mean $\mathbb{E}[y\mid x]$ (the regression function) | $\operatorname{Var}(y\mid x)=\sigma^2$ |
| absolute $|y-a|$ | the median of $p(y\mid x)$ | spread around the median |
| 0–1 (classification) | $\arg\max_c P(y=c\mid x)$ (Bayes classifier) | Bayes error $1-\max_c P(c\mid x)$ |
⚠ Clears up — loss vs risk vs empirical risk
Loss is per-example; risk is its expectation over the true distribution (what you actually care about); empirical risk is its average over your sample (what you can optimize). Training error is empirical risk; test error estimates true risk. They differ — that gap is the whole story of generalization.
⚠ Clears up — why squared loss gives the mean
$\mathbb{E}[(y-a)^2]$ is minimized at $a=\mathbb{E}[y\mid x]$ (take derivative, set to 0). So "fit with MSE" literally means "estimate the conditional mean." Absolute loss → conditional median (robust to outliers); 0–1 loss → conditional mode. Choosing a loss = choosing which statistic of $p(y\mid x)$ you estimate.
◆ Interview probe
"Your regressor has high error even with infinite data and a perfect model — why?" → The irreducible $\sigma^2=\operatorname{Var}(y\mid x)$: genuine noise / missing features. No model beats the Bayes risk; only better features (changing $\mathcal{D}$) can.
Remember We want low risk (expected loss on future data) but can only minimize empirical risk. The loss you pick decides which summary of $p(y\mid x)$ is optimal: MSE→mean, MAE→median, 0–1→mode.
Tricky interview questions 11
What's the difference between loss, risk, and empirical risk?
Loss is per-example $L(y,f(x))$. Risk is its expectation over the true distribution $\mathcal{D}$ — what you actually care about but can't compute. Empirical risk is its average over your training sample — what you actually minimize (ERM). Trap: calling training error "the risk." Training error is empirical risk; test error estimates true risk, and the gap between them is the entire generalization story.
Why does squared loss recover the conditional mean?
$\mathbb{E}[(y-a)^2\mid x]$ is a convex parabola in $a$; setting its derivative to zero gives $a=\mathbb{E}[y\mid x]$. So "fit with MSE" literally means "estimate the conditional mean." Trap: thinking the choice of loss is arbitrary — it decides which summary of $p(y\mid x)$ you target.
What does each loss make the Bayes-optimal predictor?
Squared $\to$ mean $\mathbb{E}[y\mid x]$; absolute $\to$ median of $p(y\mid x)$; 0–1 $\to$ mode $\arg\max_c P(c\mid x)$ (the Bayes classifier). Trap: assuming the "best predictor" is fixed. There is no loss-free best predictor — the optimum is defined only relative to the loss.
Your regressor has high error even with infinite data and a perfect model. Why?
The irreducible error $\sigma^2=\operatorname{Var}(y\mid x)$: genuine noise or missing features. No model beats the Bayes risk; only richer features (changing $\mathcal{D}$) lower it. Trap: blaming the model class or the optimizer — neither can remove noise that the inputs don't explain.
Write the bias–variance decomposition of expected squared error. What are the three terms?
$\mathbb{E}[(y-\hat f(x))^2]=\text{Bias}[\hat f]^2+\operatorname{Var}[\hat f]+\sigma^2$. Bias² is how far the average model (over training sets) sits from the truth; variance is how much it wobbles with the sample; $\sigma^2$ is the irreducible noise. The expectation is over both the noise in $y$ and the random draw of the training set. Trap: forgetting $\sigma^2$ and thinking error can hit zero. The noise floor lower-bounds test error no matter how good the model.
Why can't you just minimize training error?
Training error is empirical risk on the sample; what you want is true risk on future data. A flexible model can drive training error to 0 while true risk stays high — the gap is variance. Trap: treating low training error as success. It's the easiest thing to fake and says nothing about generalization.
What is the Bayes error rate?
The lowest possible error any classifier can achieve, $1-\mathbb{E}_x[\max_c P(c\mid x)]$, attained by the Bayes classifier that predicts the most probable class at each $x$. It's the classification analogue of $\sigma^2$. Trap: believing a good-enough model can beat it. Only changing the features/distribution can lower the Bayes error.
How do you diagnose whether a model suffers from high bias or high variance?
Compare training and validation error. High bias: both are poor and close together. High variance: training error is low but validation is much higher (a large gap). Trap: looking at validation error alone — a high value is ambiguous until you compare it against training error and the irreducible/Bayes baseline.
Does collecting more data reduce bias or variance?
Primarily variance — predictions stabilize and the train/validation gap shrinks. Bias barely moves, since a structurally too-simple model underfits regardless of dataset size. Trap: claiming more data fixes everything. If training and validation error are both high and close (high bias), more data won't help — you need a richer model or better features.
Why ever prefer a biased estimator?
Because $\text{MSE}=\text{bias}^2+\text{variance}$: deliberately accepting a little bias can slash variance and give lower total error. Ridge, MAP, and James–Stein all do exactly this. Trap: equating "unbiased" with "best." Unbiased is one property, not minimum error.
Why does absolute loss give a more robust predictor than squared loss?
Absolute loss targets the conditional median, while squared loss targets the mean. A squared term lets a single wild outlier dominate the sum and drag the mean, whereas the median barely moves. Trap: calling robustness a free win — the median throws away information about the magnitude of large deviations, and absolute loss has no closed form and a non-smooth gradient at zero.
06
🎯Frequentist: the data wiggles, θ stands still. Bayesian: θ wiggles, the data is fixed.
The single most-confused split in ML interviews. It comes down to one question: is the parameter $\theta$ a fixed unknown constant, or a random variable you have beliefs about? Everything else follows.
| Frequentist | Bayesian | |
|---|---|---|
| What's random | the data (θ is a fixed unknown constant) | your belief about θ (data is fixed, observed) |
| Core object | sampling distribution of an estimator $\hat\theta(D)$ | posterior $p(\theta\mid D)$ |
| Question | "what would I see if I repeated the experiment?" | "given this data, what do I believe about θ?" |
| Point estimate | MLE $\hat\theta=\arg\max p(D\mid\theta)$ | posterior mean, or MAP $\arg\max p(\theta\mid D)$ |
| Uncertainty | confidence interval (coverage of the procedure) | credible interval (probability about θ) |
| Needs | nothing but the likelihood | a prior $p(\theta)$ |
Bayes' rule is the whole Bayesian engine
$$ \underbrace{p(\theta\mid D)}_{\text{posterior}} \;=\; \frac{\overbrace{p(D\mid\theta)}^{\text{likelihood}}\;\overbrace{p(\theta)}^{\text{prior}}}{\underbrace{p(D)}_{\text{evidence}}}\;\propto\; p(D\mid\theta)\,p(\theta). $$The evidence $p(D)=\int p(D\mid\theta)p(\theta)\,d\theta$ is just the normalizer; for point estimates you can ignore it.
Predictions integrate over the posterior (the posterior predictive), averaging over parameter uncertainty instead of betting on one $\hat\theta$:
$$ p(y\mid x,D)=\int p(y\mid x,\theta)\,p(\theta\mid D)\,d\theta. $$
⚠ Clears up — confidence vs credible interval
A 95% confidence interval is a statement about the procedure: if you repeated the experiment many times, 95% of the intervals it produces would contain the true θ. It is not "95% probability θ is in this interval." A 95% credible interval is that: $P(\theta\in[a,b]\mid D)=0.95$. People constantly state the frequentist CI with the Bayesian meaning — that's the trap.
⚠ Clears up — a p-value is not P(H₀ | data)
A p-value is $P(\text{data this extreme or more}\mid H_0)$ — computed assuming the null is true. It is not the probability the null is true (that would need a prior, i.e. a Bayesian posterior). Low p = "data would be surprising if $H_0$ held," nothing more.
The bridge (why you don't have to pick a side)
- MLE = MAP with a flat prior. Add a prior and the Bayesian point estimate becomes a regularized MLE (ch4).
- Lots of data ⇒ they agree. The posterior concentrates at the MLE and becomes Gaussian (Bernstein–von Mises); the prior washes out.
- Regularization is a prior. L2 = Gaussian prior, L1 = Laplace prior. So "frequentist with a penalty" ≈ "Bayesian MAP."
- In practice most ML is frequentist (fit by MLE/ERM) with Bayesian-flavored regularization; full Bayesian (integrate over θ) shows up in small-data, calibration, and bandits/active learning.
◆ Interview probe
"Is logistic regression frequentist or Bayesian?" → The model is just a likelihood $p(y\mid x,\theta)$. Fitting it by MLE is frequentist; adding L2 makes it MAP with a Gaussian prior (a Bayesian point estimate); putting a full posterior on the weights makes it Bayesian logistic regression. The model isn't either — how you treat θ is.
Remember Frequentist = randomness in the data, θ fixed → MLE + confidence intervals. Bayesian = randomness in your belief, prior + posterior → MAP/posterior-mean + credible intervals. Bridge: MAP = MLE + log-prior.
Tricky interview questions 12
Explain the difference between the Frequentist and Bayesian approaches.
Frequentists treat $\theta$ as a fixed unknown and randomness as long-run frequency, reporting point estimates, p-values, and confidence intervals. Bayesians treat $\theta$ as a random variable with a prior, update to a posterior via Bayes' rule, and report credible intervals. Trap: Saying a credible interval and a confidence interval mean the same thing — only the credible interval supports "95% probability $\theta$ is in here."
What is the difference between MLE and MAP estimation?
MLE maximizes the likelihood $p(D\mid\theta)$; MAP maximizes the posterior $p(\theta\mid D)\propto p(D\mid\theta)\,p(\theta)$, so it adds a prior. MAP = MLE with a flat prior; a zero-mean Gaussian prior gives L2, a Laplace prior gives L1. Trap: Saying MAP is "always better," or confusing MAP (the posterior mode, one point) with full Bayes (the whole posterior). And forgetting MLE is just MAP with a uniform prior.
What is the difference between probability and likelihood?
In probability, $\theta$ is fixed and you ask how probable the data is: $p(\text{data}\mid\theta)$. Likelihood fixes the observed data and reads the same expression as a function of $\theta$: $L(\theta)=p(\text{data}\mid\theta)$. Trap: Treating the likelihood as a probability distribution over $\theta$ — it does not integrate to 1 over $\theta$.
Confidence interval vs credible interval — what's the real difference?
A 95% confidence interval is a property of the procedure: repeat the experiment many times and 95% of the intervals it builds contain the true $\theta$. A 95% credible interval is the Bayesian one: $P(\theta\in[a,b]\mid D)=0.95$. Trap: Saying "there's a 95% probability $\theta$ is in this specific CI." Once computed, a fixed CI either contains $\theta$ or doesn't; the 95% lives in the method, not the interval.
What is a p-value, and is it $P(H_0\mid\text{data})$?
A p-value is $P(\text{data this extreme or more}\mid H_0)$ — computed assuming the null is true. It says the data would be surprising if $H_0$ held; nothing about effect size. Trap: Calling it "the probability the null is true" or "the probability the result is due to chance." That would be $P(H_0\mid\text{data})$, which needs a prior — a Bayesian posterior, not a p-value.
State Bayes' theorem and solve the rare-disease test (0.1% prevalence, 99% sensitivity, 99% specificity).
$P(\theta\mid D)=\frac{p(D\mid\theta)\,p(\theta)}{p(D)}$. Here $P(\text{disease}\mid +)=\frac{0.99\cdot0.001}{0.99\cdot0.001+0.01\cdot0.999}\approx 9\%$, because false positives from the huge healthy population swamp the few true positives. Trap: Base-rate fallacy — ignoring the tiny prior and answering ~99%. The answer is low precisely because the disease is rare.
Is logistic regression frequentist or Bayesian?
Neither — the model is just a likelihood $p(y\mid x,\theta)$. Fitting it by MLE is frequentist; adding L2 makes it MAP with a Gaussian prior (a Bayesian point estimate); putting a full posterior on the weights makes it Bayesian logistic regression. Trap: Thinking the model "is" one camp. The model isn't either — how you treat $\theta$ decides.
MLE, MAP, and full Bayes — what's the difference between the three?
All start from the same likelihood. MLE = peak of the likelihood (no prior). MAP = peak of the posterior (likelihood × prior), a single point = MLE + regularizer. Full Bayes = keep the whole posterior and integrate over it for predictions. Trap: Calling MAP "Bayesian inference." MAP is a single point (the mode); it throws away the rest of the posterior, so it gives no uncertainty the way full Bayes does.
How does a Bayesian make a prediction — and how is it different from plugging in $\hat\theta$?
It integrates over the posterior (the posterior predictive): $p(y\mid x,D)=\int p(y\mid x,\theta)\,p(\theta\mid D)\,d\theta$, averaging over parameter uncertainty instead of betting on one $\hat\theta$. Trap: Treating "Bayesian prediction" as just predicting with the MAP estimate — that's a point plug-in and ignores parameter uncertainty, which is the whole point of going Bayesian.
When do the frequentist and Bayesian answers agree, and when do they diverge?
With lots of data they agree: the likelihood dominates, the posterior concentrates at the MLE and becomes Gaussian (Bernstein–von Mises), and the prior washes out. They diverge in small-data, high-uncertainty regimes where the prior still matters — calibration, bandits, active learning. Trap: Assuming the prior always biases you. With enough data its influence vanishes; it only bites when data is scarce.
How is regularization secretly Bayesian?
A regularizer is a negative log-prior: L2 = Gaussian prior $\mathcal{N}(0,\tau^2 I)$, L1 = Laplace prior. So "frequentist fit with a penalty" is exactly MAP, and a tighter prior (smaller $\tau$) means a larger $\lambda$ — more shrinkage. Trap: Thinking ridge/lasso are purely frequentist tricks. They're Bayesian MAP estimates in disguise; the penalty strength is how strongly you believe weights sit near zero.
Does zero correlation imply independence, and how does that connect to "uncorrelated vs independent"?
No. Independence implies zero correlation, but not the reverse — correlation only sees linear association. E.g. $Y=X^2$ with $X$ symmetric about 0 is uncorrelated yet fully dependent. The exception: jointly Gaussian variables, where uncorrelated does imply independent. Trap: Asserting uncorrelated means independent in general — it's true only in special cases like the joint Gaussian.
07
🎯Loss is −log likelihood; the penalty is −log prior. MAP = MLE wearing a prior.
The most useful unification in the notes: every common loss is a negative log-likelihood, and every common regularizer is a negative log-prior. Once you see this, you can read off the loss from the noise model and the penalty from the prior.
MLE — fit by maximizing the likelihood
$$ \hat\theta_{\text{MLE}}=\arg\max_\theta \prod_i p(y_i\mid x_i,\theta)=\arg\min_\theta \sum_i -\log p(y_i\mid x_i,\theta). $$Maximize a product of probabilities ⇔ minimize a sum of negative log-likelihoods (the loss).
The dictionary — choose the noise model, read off the loss:
| Assume $p(y\mid x)$ is… | −log-likelihood = | i.e. the loss |
|---|---|---|
| Gaussian $\mathcal{N}(f_\theta(x),\sigma^2)$ | $\tfrac{1}{2\sigma^2}(y-f_\theta(x))^2+c$ | squared error (MSE) |
| Bernoulli $p=\sigma(f_\theta(x))$ | $-[y\log p+(1-y)\log(1-p)]$ | cross-entropy (log loss) |
| Categorical (softmax) | $-\log p_{\text{true class}}$ | multiclass cross-entropy |
| Laplace | $|y-f_\theta(x)|/b+c$ | absolute error (MAE) |
| Poisson | $f_\theta(x)-y\log f_\theta(x)+c$ | Poisson loss |
MAP — add a prior, get a regularizer for free
$$ \hat\theta_{\text{MAP}}=\arg\max_\theta\; p(D\mid\theta)\,p(\theta)=\arg\min_\theta\Big[\underbrace{\textstyle\sum_i -\log p(y_i\mid x_i,\theta)}_{\text{loss}}\;+\;\underbrace{(-\log p(\theta))}_{\text{regularizer}}\Big]. $$
| Prior $p(\theta)$ | −log-prior = | i.e. the penalty |
|---|---|---|
| Gaussian $\mathcal{N}(0,\tau^2 I)$ | $\tfrac{1}{2\tau^2}\lVert\theta\rVert_2^2+c$ | L2 / ridge, $\lambda=\tfrac{\sigma^2}{\tau^2}$ |
| Laplace (mean 0) | $\tfrac{1}{b}\lVert\theta\rVert_1+c$ | L1 / lasso (→ sparsity) |
| flat / improper | constant | no penalty → MAP = MLE |
So a tighter prior (smaller $\tau$) ⇒ larger $\lambda$ ⇒ more shrinkage. Regularization strength is literally how strongly you believe weights are near zero.
◆ Interview probe — derive ridge = Gaussian-prior MAP
Likelihood $y_i\sim\mathcal{N}(\theta^\top x_i,\sigma^2)$, prior $\theta\sim\mathcal{N}(0,\tau^2I)$. Then $-\log p(\theta\mid D)=\tfrac{1}{2\sigma^2}\sum_i (y_i-\theta^\top x_i)^2+\tfrac{1}{2\tau^2}\lVert\theta\rVert^2+c$. Multiply by $2\sigma^2$: minimize $\sum_i(y_i-\theta^\top x_i)^2+\lambda\lVert\theta\rVert^2$ with $\lambda=\sigma^2/\tau^2$. That's ridge. Laplace prior → the same with $\lVert\theta\rVert_1$ = lasso.
⚠ Clears up — MLE vs MAP vs full Bayes
All three start from the same likelihood. MLE = peak of the likelihood (no prior). MAP = peak of the posterior (likelihood × prior) — a single point, = MLE + regularizer. Full Bayes = keep the whole posterior and integrate over it for predictions (uncertainty-aware, no overfitting to a point, but needs the integral). MAP is the bridge: a Bayesian object computed like a regularized frequentist fit.
Remember loss $=-\log$ likelihood; regularizer $=-\log$ prior; MAP $=$ MLE $+$ regularizer. Gaussian noise→MSE, Bernoulli→cross-entropy; Gaussian prior→L2, Laplace prior→L1.
Tricky interview questions 12
What is the difference between MLE and MAP estimation?
MLE maximizes the likelihood $P(D\mid\theta)$; MAP maximizes the posterior $P(\theta\mid D)\propto P(D\mid\theta)\,P(\theta)$, so it just adds a prior. In log-space MAP = MLE + (−log prior), i.e. a regularized MLE: a zero-mean Gaussian prior gives L2/ridge, a Laplace prior gives L1/lasso. Trap: calling MAP "always better," or confusing MAP (a single point — the posterior mode) with full Bayes (the whole posterior). And MLE is just MAP with a flat prior.
What is the difference between probability and likelihood?
Probability fixes $\theta$ and asks how probable the data is: $P(\text{data}\mid\theta)$. Likelihood fixes the observed data and reads the same expression as a function of $\theta$: $L(\theta)=P(\text{data}\mid\theta)$. Trap: treating the likelihood as a distribution over $\theta$ — it does not integrate to 1 over $\theta$.
Where does squared-error loss actually come from?
From assuming Gaussian noise: $y\mid x\sim\mathcal{N}(f_\theta(x),\sigma^2)$. Its negative log-likelihood is $\tfrac{1}{2\sigma^2}(y-f_\theta(x))^2+c$, so minimizing NLL = minimizing squared error. MSE literally estimates the conditional mean. Trap: thinking MSE is an arbitrary default — it is the Gaussian-noise MLE, and that's exactly why it's fragile to outliers.
Why is cross-entropy the right loss for classification (vs MSE)?
Cross-entropy is the Bernoulli/categorical negative log-likelihood, $-[y\log p+(1-y)\log(1-p)]$ — the principled MLE objective. With a sigmoid it stays convex and keeps a strong residual gradient $(p-y)x$ exactly when the model is confidently wrong; sigmoid+MSE is non-convex with vanishing gradients there. Trap: saying MSE "can't be optimized" — it can; the real issues are non-convexity, dead gradients, and that it's the wrong noise model.
Derive ridge regression as a MAP estimate.
Take $y_i\sim\mathcal{N}(\theta^\top x_i,\sigma^2)$ and prior $\theta\sim\mathcal{N}(0,\tau^2 I)$. Then $-\log p(\theta\mid D)=\tfrac{1}{2\sigma^2}\sum_i(y_i-\theta^\top x_i)^2+\tfrac{1}{2\tau^2}\lVert\theta\rVert_2^2+c$. Multiply by $2\sigma^2$: minimize $\sum_i(y_i-\theta^\top x_i)^2+\lambda\lVert\theta\rVert_2^2$ with $\lambda=\sigma^2/\tau^2$. Trap: dropping the $\sigma^2/\tau^2$ ratio — a tighter prior (smaller $\tau$) means larger $\lambda$, i.e. more shrinkage.
Why does an L1 (Laplace) prior give sparsity but L2 (Gaussian) does not?
L1's penalty $\tfrac{1}{b}\lVert\theta\rVert_1$ has a constant-magnitude gradient that keeps pushing small weights to exactly zero (its constraint ball has corners on the axes); L2's penalty $\propto\lVert\theta\rVert_2^2$ has a gradient that shrinks as weights shrink, so it pulls smoothly toward zero but never reaches it. Trap: saying L2 zeroes out features. Only L1 produces exact zeros / feature selection.
MLE vs MAP vs full Bayes — what's the difference?
All three share the same likelihood. MLE = peak of the likelihood (no prior). MAP = peak of the posterior = likelihood × prior, a single point = MLE + regularizer. Full Bayes keeps the whole posterior and integrates over it for predictions (the posterior predictive), so it's uncertainty-aware and doesn't bet on one $\hat\theta$. Trap: calling MAP "Bayesian inference" — it's a point estimate computed like a regularized frequentist fit, not a posterior.
Is the MLE unbiased? Why ever prefer a biased estimator?
Not in general — MLE is consistent and asymptotically efficient, but can be biased in finite samples (e.g. the Gaussian-variance MLE divides by $n$, biased low by the factor $\tfrac{n-1}{n}$). You prefer a biased estimator when it lowers MSE: since $\text{MSE}=\text{bias}^2+\text{variance}$, ridge/MAP trade a little bias for a big variance cut. Trap: equating "unbiased" with "best." Unbiased is not the same as low MSE.
Derive the MLE for the Bernoulli parameter $p$.
With $k$ successes in $n$ trials, $\ell(p)=k\log p+(n-k)\log(1-p)$. Set $\ell'(p)=\tfrac{k}{p}-\tfrac{n-k}{1-p}=0\Rightarrow\hat p=k/n$, the sample proportion; $\ell''<0$ confirms a max. Trap: not taking the log first (products are painful to differentiate), or jumping to $k/n$ without verifying it's a maximum.
Explain the Frequentist vs Bayesian split, and how MLE/MAP bridge it.
Frequentist: $\theta$ is a fixed unknown, the data is random → MLE + confidence intervals. Bayesian: $\theta$ is random with a prior, the data is fixed → posterior, MAP/posterior-mean + credible intervals. The bridge: MLE = MAP with a flat prior, and adding a regularizer = adding a prior, so "frequentist with a penalty" $\approx$ Bayesian MAP. Trap: saying a confidence interval and a credible interval mean the same thing — only the credible interval supports "95% probability $\theta$ is in here."
How does the regularization strength $\lambda$ map onto a prior belief?
For ridge, $\lambda=\sigma^2/\tau^2$ where $\tau^2$ is the prior variance on the weights. A small $\tau$ (you strongly believe weights are near zero) gives a large $\lambda$ and heavy shrinkage; $\tau\to\infty$ (flat prior) gives $\lambda\to0$ and recovers MLE. Trap: thinking $\lambda$ is a free dial with no meaning — it literally encodes how tightly you believe weights cluster at zero.
You assume Poisson noise — what loss do you get, and when do you use it?
Poisson NLL is $f_\theta(x)-y\log f_\theta(x)+c$ (the Poisson/log loss), used for non-negative count targets with the rate $\lambda=f_\theta(x)>0$. It's the same "pick a noise model, read off the loss" recipe as Gaussian→MSE and Bernoulli→cross-entropy. Trap: fitting counts with MSE — that silently assumes Gaussian, symmetric, constant-variance noise, which counts violate (variance grows with the mean).
08
🎯Bias = wrong on average. Variance = jumpy. Noise = unfixable. Data kills variance, not bias.
The reason you can't just minimize training error. Test error splits into three pieces you trade off by choosing model complexity. This is the exact decomposition from ESL §7.3 — derived, not asserted.
Setup & the decomposition
Fix a test point $x$. The world generates $y=f(x)+\varepsilon$ with $\mathbb{E}[\varepsilon]=0,\ \operatorname{Var}(\varepsilon)=\sigma^2$. You train on a random dataset $D$ and get $\hat f_D$. Average the squared error over both the noise and the draw of $D$:
$$ \mathbb{E}\big[(y-\hat f_D(x))^2\big]=\underbrace{\sigma^2}_{\text{noise}}+\underbrace{\big(\bar f(x)-f(x)\big)^2}_{\text{bias}^2}+\underbrace{\mathbb{E}\big[(\hat f_D(x)-\bar f(x))^2\big]}_{\text{variance}},\quad \bar f(x)=\mathbb{E}_D[\hat f_D(x)]. $$
noise σ²irreducible — the Bayes floor; no model removes it.
bias²how far the average model is from the truth → underfitting / too-simple class.
variancehow much the model wobbles with the training sample → overfitting / too-flexible class.
The derivation (one screen)
Add and subtract $\bar f(x)$, and use that $\varepsilon$ and $\hat f_D$ are independent so cross-terms vanish:
$$ \mathbb{E}[(y-\hat f)^2]=\mathbb{E}[(f+\varepsilon-\hat f)^2]=\sigma^2+\mathbb{E}[(f-\hat f)^2] $$noise splits off because $\varepsilon\perp\hat f$ and $\mathbb{E}[\varepsilon]=0$.
$$ \mathbb{E}[(f-\hat f)^2]=\mathbb{E}[(\underbrace{f-\bar f}_{\text{const}}+\underbrace{\bar f-\hat f}_{\text{mean }0})^2]=(f-\bar f)^2+\mathbb{E}[(\hat f-\bar f)^2]=\text{bias}^2+\text{variance}. $$
⚠ Clears up — the four things people get wrong
(1) It's an average over training sets, not one model. Bias is about the mean prediction $\bar f$; variance is its spread as $D$ changes. (2) More data ↓ variance, not bias. A linear model on a curved truth stays biased no matter how much data — you need a richer class to cut bias. (3) Capacity trades them: complexity ↑ ⇒ bias ↓, variance ↑ ⇒ U-shaped test error; the minimum is the sweet spot. (4) Training vs test: a flexible model drives training error toward 0 (low apparent bias) while test error is bias²+variance+σ² — the gap is variance.
⚠ Clears up — does it apply to classification?
The clean 3-term split is a squared-loss (regression) result. For 0–1 loss there's an analogous decomposition (Domingos 2000) but the terms interact and bias/variance combine non-additively; the intuition (simple→biased, flexible→high-variance) still holds, the tidy algebra doesn't.
Double descent — why huge models break the U-curve
Classical wisdom: past the sweet spot, more capacity = more variance = worse test error. Modern reality: keep growing past the interpolation threshold (enough parameters to hit ~0 training error, params ≈ $n$) and test error spikes there, then descends again.
why the spikeat params ≈ $n$ there's exactly one interpolating fit; it's wild between points → huge variance.
why the 2nd dropwith many more params there are many zero-error fits; SGD's implicit bias picks a low-norm / smooth one → variance falls again.
takeawayover-parameterized nets live in the second-descent regime; "more params" can help even at zero train error.
◆ Interview probe
"Val error is high and adding data barely helps — bias or variance?" → bias / underfitting: the average model is wrong, so more samples of the same class won't help; enrich the model or features. (If more data closed the gap, it was variance.) Follow-up: "How do you cut each?" Variance ↓: more data, regularization, bagging, simpler model. Bias ↓: richer model/features, boosting, less regularization.
Remember test error = σ² (noise) + bias² (underfit) + variance (overfit). Data & regularization cut variance; capacity & features cut bias. Double descent = a second fall in error after the model is big enough to memorize.
Tricky interview questions 12
Explain the bias-variance tradeoff and how it relates to under/overfitting.
Bias = error from too-simple assumptions (underfits, misses real patterns); variance = sensitivity to the specific training sample (overfits, fits noise). Complexity up ⇒ bias down, variance up, so you tune complexity to the sweet spot of total test error. Trap: claiming you can drive both to zero — with a fixed class and dataset, cutting one usually raises the other.
Write the bias-variance decomposition of expected squared error. What are the three terms?
$\mathbb{E}[(y-\hat f(x))^2]=\text{Bias}[\hat f]^2+\text{Var}[\hat f]+\sigma^2$, where $\sigma^2$ is irreducible noise in $y$. The expectation is over both the label noise and the random training set. Trap: dropping $\sigma^2$ or thinking error can hit zero — the noise floor $\sigma^2$ lower-bounds test error no matter how good the model.
How do you diagnose high bias vs high variance from train/val error?
High bias: training error high and val error close to it (both poor). High variance: training error low but val error much higher — a big gap. Trap: looking only at val error. A high val error alone is ambiguous; you must compare it to training error (and the Bayes baseline) to know which problem you have.
Your model has high variance. List concrete ways to reduce it.
More data, regularization (L1/L2, dropout, early stopping), fewer features / simpler model, and bagging/ensembling; pick complexity by cross-validation. Trap: suggesting a more complex model or more features — those fight variance the wrong way and worsen overfitting; they are bias fixes.
Your model has high bias. How do you reduce it?
Increase capacity (more expressive model, more/better features, interaction or polynomial terms, deeper net), reduce regularization, train longer; boosting also cuts bias. Trap: adding more data — it doesn't fix bias. A model too simple to capture the pattern stays wrong no matter how many points you feed it.
How does regularization strength $\lambda$ move you along the bias-variance curve?
Bigger $\lambda$ shrinks weights, simplifying the model: bias up, variance down. $\lambda\to 0$ recovers the unregularized (low-bias, high-variance) fit; $\lambda\to\infty$ drives weights to zero (very high bias). Pick $\lambda$ by CV. Trap: thinking bigger $\lambda$ is always better — too much underfits; there's an optimal $\lambda$, not "more is better."
In k-NN, how does k affect bias and variance?
Small k (k=1) = low bias, high variance (hugs the data, noise-sensitive). Large k = high bias, low variance (smooths over many neighbors). k is the complexity knob. Trap: getting the direction backward — larger k is simpler, not more complex.
Why does bagging (e.g. random forests) reduce variance but not bias?
It averages many high-variance models trained on bootstrap samples; averaging roughly independent predictors cuts variance (toward $\text{Var}/N$ when uncorrelated) while each base learner keeps the same bias. RFs decorrelate trees by feature sampling to push variance lower. Trap: claiming bagging reduces bias — averaging same-bias models doesn't change bias, and tree correlation caps how far variance drops.
Why does boosting reduce bias, and what's its risk?
It fits weak learners sequentially, each correcting the residual errors of the ensemble so far, progressively cutting bias. Risk: overfitting (rising variance) if boosted too long without shrinkage/early stopping. Trap: treating boosting and bagging as interchangeable — bagging attacks variance with parallel models, boosting attacks bias with sequential ones, and boosting can overfit.
What is double descent, and how does it break the classical U-curve?
Past the interpolation threshold (params ≈ $n$, where training error first hits 0), test error peaks then descends a second time — huge over-parameterized nets generalize well despite zero training error. Trap: asserting the U-curve is universal. In the over-parameterized regime, adding capacity can lower test error again.
Why does test error spike exactly at the interpolation threshold, then fall again?
At params ≈ $n$ there's essentially one interpolating fit; it's forced to wiggle wildly between points → huge variance. With many more params there are many zero-error fits, and SGD's implicit bias picks a low-norm / smooth one → variance falls again. Trap: attributing the second descent to less noise — $\sigma^2$ is fixed; it's variance dropping via implicit regularization.
Does the clean 3-term decomposition apply to classification?
The additive $\text{bias}^2+\text{variance}+\sigma^2$ split is a squared-loss (regression) result. For 0–1 loss there's an analogous decomposition (Domingos 2000), but the terms interact non-additively. Trap: writing the tidy additive formula for 0–1 loss — the intuition (simple→biased, flexible→high-variance) holds, the algebra doesn't.
09
🎯Least squares is the line a Gaussian would draw.
Linear regression is the line a Gaussian would draw — it is exactly the maximum-likelihood fit when noise is normal. Everything else (gradient descent, the normal equation, regularization) is just how you find that line efficiently.
THE MODEL IN ONE SHOT
Predict a number as a weighted sum of the inputs. Absorb the intercept by tacking a constant 1 onto every feature vector, so the weight vector θ handles the bias automatically.
$$\hat y = \theta^\top x = \sum_{j=1}^d \theta_j x_j$$prediction is a dot product of weights and features
Fit it by minimizing the half-sum of squared residuals — the least-squares cost:
$$J(\theta) = \tfrac{1}{2}\|X\theta - y\|^2$$half the total squared error over all training examples
The ½ is cosmetic — it cancels the 2 in the derivative so the gradient looks clean.
TWO WAYS TO MINIMIZE J
Closed form (normal equation). The cost is a convex bowl, so set its gradient to zero and solve directly.
$$\theta^* = (X^\top X)^{-1} X^\top y$$exact solution — no iterations, no learning rate
Cost: ≈ O(nd²) to build X⊤X, then O(d³) to invert. Fine for a few hundred features; painful for thousands.
Gradient descent (LMS rule). On each example, nudge θ in the direction that reduces the error:
$$\theta := \theta + \alpha\,(y_i - \hat y_i)\,x_i$$step size = learning rate × residual × feature vector
Big error ⟹ big step. Small error ⟹ tiny step. Because J is convex, any small enough α converges to the global minimum.
| Variant | Update uses | Trade-off |
|---|---|---|
| Batch GD | all n examples | exact direction; O(nd) per step |
| SGD / mini-batch | 1 or m examples | noisy but fast; needs α decay to settle |
ASSUMPTIONS (LINE ≠ TRUTH)
Linearitytrue relationship is linear in the features
Independenceeach training error is independent of the others
Homoscedasticityerror variance is the same across all x
Normality of errorsneeded for valid p-values and CIs, not for predictions
No multicollinearityif predictors are correlated, coefficients become unstable
The key interviewer gotcha: it is the residuals that must be normal — not the features, not y itself.
R² AND ADJUSTED R²
R² = 1 − SSres/SStot, the fraction of target variance the model explains. Adding any feature can only increase plain R² — even noise. Adjusted R² penalizes extra parameters, so it can drop when a new feature adds nothing. Always use adjusted R² when comparing models of different sizes.
⚠ Clears up — "normal" means the residuals, not the features A common slip: "linear regression assumes the features are normally distributed." Wrong. The normality assumption is on the errors, and even then it only matters for inference (confidence intervals, p-values). The point-prediction formula works regardless of the feature distribution.
◆ Interview probe When would you pick gradient descent over the normal equation? → When d is large (say > 10,000 features), forming and inverting X⊤X costs O(d³), which is prohibitive. Gradient descent scales to millions of features and examples. Also, the normal equation requires X⊤X to be invertible — it fails under perfect multicollinearity or when d > n.
Remember Least squares is the maximum-likelihood line under Gaussian noise — the normal equation solves it exactly in one shot, gradient descent scales when that becomes too expensive.
Tricky interview questions 10
What are the assumptions of linear regression, and which ones matter for predictions vs. inference?
Linearity, independence of errors, homoscedasticity (constant error variance), and normality of residuals. Linearity and independence affect predictions directly; normality and homoscedasticity mainly affect the validity of standard errors, p-values, and confidence intervals, not the coefficient point estimates. Trap: saying the features must be normally distributed — it is the residuals that must be normal, not the inputs.
What does the normal equation give you, and when would you prefer gradient descent?
The normal equation θ = (X⊤X)⁻¹X⊤y gives the exact OLS solution with no iterations. Building X⊤X is O(nd²) and inverting it is O(d³), so for large d (thousands of features) gradient descent is far cheaper. The normal equation also fails when X⊤X is singular — under perfect multicollinearity or when d > n. Trap: assuming the normal equation always wins because it's "exact" — it scales cubically in features and breaks under collinearity.
Why is least squares equivalent to maximum likelihood estimation, and under what assumption?
If you model errors as i.i.d. Gaussian (ε ~ N(0, σ²)), the log-likelihood factors into a sum of squared residuals. Maximizing the log-likelihood is therefore identical to minimizing the sum of squared residuals — so OLS is exactly the MLE. Trap: thinking MLE always gives least squares. The Gaussian noise assumption is what makes the squared-error cost the right likelihood; non-Gaussian noise yields different loss functions.
What is R-squared, and why can't it decrease when you add a feature?
R² = 1 − SS_res/SS_tot measures the fraction of variance explained. Because OLS always minimizes SS_res, adding a feature can only keep or lower it — the model can set a new coefficient to zero if the feature is useless, matching the old fit exactly. So plain R² is monotone in the number of features. Adjusted R² penalizes complexity and can drop. Trap: using plain R² to compare models of different sizes — only adjusted R² can signal that a new feature doesn't help.
What is multicollinearity, how do you detect it, and what does it actually hurt?
High correlation among predictors inflates the variance of coefficient estimates, making signs and magnitudes swing wildly when the feature set changes. Detect it with the variance inflation factor (VIF > 5–10 is a warning sign) or a correlation matrix. Mitigate with feature removal, PCA, or regularization. Trap: saying multicollinearity hurts predictive accuracy or R² — it mainly destabilizes individual coefficients and their standard errors; overall predictions can remain fine.
Why does L1 regularization produce sparse weights but L2 doesn't?
L1 adds λΣ|w|, whose constraint region is a diamond with corners on the axes; the loss contours typically touch a corner first, zeroing out some weights exactly. L2 adds λΣw², a circular/spherical constraint that shrinks all weights smoothly toward (but never exactly to) zero. Trap: saying L2 also zeroes out coefficients. Only L1 yields exact zeros (built-in feature selection); L2 handles correlated features more gracefully and is differentiable everywhere.
What is the LMS update rule, and why does the step size scale with the residual?
The LMS (least-mean-squares) rule is θ := θ + α(y_i − θ⊤x_i)x_i. The gradient of the squared error for a single example is the residual times x_i, so a gradient-descent step naturally scales with how wrong the prediction is — big error, big correction; correct prediction, zero update. Trap: confusing LMS (single-example gradient step) with the full batch gradient update, which sums over all n examples per step.
What assumptions of linear regression does logistic regression drop?
Logistic regression does not assume normally distributed residuals or homoscedasticity. Its linearity assumption is on the log-odds (logit), not on the raw probability. It still assumes independent observations and little multicollinearity. Trap: carrying over linear regression's error-normality and constant-variance requirements into logistic regression — both are absent.
Why can't we just use linear regression for binary classification?
Linear regression produces outputs outside [0,1], so they can't be read as probabilities. It also assumes Gaussian, homoscedastic errors, which a 0/1 outcome violates, and leverage points can shift the decision boundary arbitrarily. Trap: thinking the only problem is output range. Even after clipping, the variance and normality assumptions are violated, and extreme points distort the boundary in ways a proper probabilistic model avoids.
When would you use SGD instead of batch gradient descent for fitting a linear model?
When n is large, computing the full-data gradient costs O(nd) per step, which is slow. SGD uses one (or a mini-batch of) example(s), so each step is much cheaper, and the noisy updates often generalize better. The trade-off is noisy convergence: α must decay over time for SGD to settle, whereas batch GD converges smoothly with a fixed small α. Trap: claiming SGD "doesn't converge" — it converges in expectation with an appropriate decaying learning rate, and with a fixed α it oscillates in a neighborhood of the optimum.
10
🎯Sigmoid squashes the score; cross-entropy punishes confident lies.
Take a linear score, squash it through an S-curve, read it as a probability — that's logistic regression in one breath. The trick is so clean it became the backbone of every modern classifier and neural network output layer.
THE MODEL: SCORE → PROBABILITY
Compute the usual linear score $z = \theta^\top x$, then wrap it in the sigmoid to get a probability between 0 and 1. The score $z$ is the log-odds — the log of how much more likely class 1 is than class 0. The sigmoid undoes that log to give you the probability directly.
$$p = \sigma(\theta^\top x) = \frac{1}{1+e^{-\theta^\top x}}$$probability of class 1 given features x
$z = \theta^\top x$log-odds (logit); linear, unbounded
$\sigma(z)$S-shaped; $\sigma(0)=0.5$, $\sigma(\pm\infty)\to\{0,1\}$
classifypredict 1 when $p \ge 0.5$, i.e. $z \ge 0$ — a hyperplane boundary
THE LOSS: PUNISH CONFIDENT LIES
Labels are Bernoulli draws, so we fit by maximum likelihood. The negative log-likelihood is cross-entropy. It is convex in $\theta$ — gradient descent reaches the unique global minimum. Say the true label is 1 and the model outputs $p = 0.01$: $-\log(0.01) \approx 4.6$, a steep penalty. MSE would only see $(1-0.01)^2 \approx 1$ — a shrug.
$$J(\theta) = -\sum_i \bigl[y_i \log p_i + (1-y_i)\log(1-p_i)\bigr]$$sum of penalties for each wrong (or confidently-wrong) prediction
LINEAR VS. LOGISTIC AT A GLANCE
| Linear regression | Logistic regression | |
|---|---|---|
| Output | any real number | probability in (0,1) |
| Link | identity | sigmoid |
| Loss | MSE / Gaussian NLL | cross-entropy / Bernoulli NLL |
| Closed form? | yes (normal equation) | no — iterative (GD, IRLS) |
| Assumption | linear in $y$ | linear in log-odds |
COEFFICIENTS & REGULARIZATION
Each $\theta_j$ shifts the log-odds by $\theta_j$ per unit of $x_j$; $e^{\theta_j}$ is the odds ratio. That is not a linear shift in probability — the same $\theta_j$ has a bigger effect on probability near 0.5 than near the extremes. Add L2 (Ridge) to shrink all weights smoothly; add L1 (Lasso) to zero some out entirely, because L1's diamond-shaped constraint region has corners on the axes.
⚠ Clears up — "logistic regression assumes normality" Logistic regression carries none of linear regression's error-normality or homoscedasticity baggage. Its linearity assumption is only on the log-odds, not on the raw probability or the residuals.
◆ Interview probe Why can't we just use MSE for binary classification? → MSE composed with the sigmoid is non-convex (spurious local minima, vanishing gradients on confident errors). Cross-entropy is the principled MLE objective and stays convex — clean optimization, no local traps.
Remember Logistic regression is linear regression with a sigmoid output and a cross-entropy loss — same linear core, Bernoulli noise model, convex objective, hyperplane boundary.
Tricky interview questions 10
What is the core difference between linear and logistic regression?
Linear regression predicts a continuous output and is fit by minimizing squared error. Logistic regression passes the same linear score through a sigmoid to bound output in (0,1), fit by Bernoulli MLE (cross-entropy). Both are linear models — but logistic is linear in the log-odds, not in the probability. Trap: Stopping at "logistic is for classification." The deeper answer is that logistic regression is a GLM with a Bernoulli noise model; it is linear in log-odds space.
Why can't we just use linear regression for binary classification?
Linear regression produces unbounded outputs that can't be read as probabilities. Even after clipping, it assumes constant-variance Gaussian errors — which a 0/1 label clearly violates — and extreme-valued points can pull the decision boundary in harmful ways. Trap: Thinking the only problem is output range. Non-constant variance and non-normal errors are equally serious, and MSE with the sigmoid is also non-convex.
Why does logistic regression use cross-entropy instead of MSE?
Cross-entropy is exactly the Bernoulli negative log-likelihood, so it is the principled MLE objective. With the sigmoid, it is convex in the weights (unique global minimum, no vanishing-gradient plateau), while MSE composed with sigmoid is non-convex and produces near-zero gradients when the model is confidently wrong. Trap: Saying MSE "can't be optimized." It can; the real issues are non-convexity and that log-loss, not MSE, is the correct probabilistic objective.
Is there a closed-form solution for logistic regression?
No. Unlike linear regression's normal equation $w=(X^\top X)^{-1}X^\top y$, the logistic MLE has no closed form and is solved iteratively via gradient descent or Newton's method (IRLS). Trap: Claiming logistic regression has a normal-equation solution like OLS. Least squares is not the MLE for a Bernoulli outcome.
How do you interpret a logistic regression coefficient?
A one-unit increase in $x_j$ changes the log-odds by $\theta_j$, holding other features fixed. Exponentiating gives the odds ratio: $e^{\theta_j}$. An odds ratio of 1.14 means roughly 14% more odds per unit increase. Trap: Saying the coefficient changes the probability by a fixed amount. The probability shift is nonlinear and depends on the baseline — an odds ratio of 2 doubles the odds, not the probability.
What does the decision boundary of logistic regression look like, and what are its limits?
The decision boundary is a hyperplane: the set of points where $\theta^\top x = 0$ (predicted probability = 0.5). It is strictly linear in the input features, so logistic regression can only separate approximately linearly-separable classes without adding polynomial or interaction terms. Trap: Assuming logistic regression can fit curved boundaries by itself — it cannot without explicit feature engineering.
What is the difference between L1 and L2 regularization, and why does L1 produce sparse weights?
L1 adds $\lambda\sum|w_j|$ and L2 adds $\lambda\sum w_j^2$. L1 drives coefficients to exact zero because its diamond-shaped constraint region has corners on the axes; the optimum typically lands there. L2 shrinks all weights smoothly toward zero but rarely exactly to zero. Trap: Saying L2 also zeros out coefficients. Only L1 yields exact sparsity; L2 handles correlated features more gracefully and is differentiable everywhere.
What are the assumptions of logistic regression, and how do they differ from linear regression's?
Logistic regression assumes linearity between predictors and the log-odds, independent observations, and little multicollinearity. It does NOT assume normally distributed residuals or homoscedasticity. Linear regression additionally requires both of those for valid inference. Trap: Carrying over linear regression's normality and constant-variance assumptions into logistic regression — they don't apply.
For an imbalanced binary problem, what metric and decision threshold should you use?
Avoid accuracy (a majority classifier looks good). Use precision, recall, F1, and PR-AUC. Tune the decision threshold to the relative cost of false positives vs. false negatives rather than defaulting to 0.5. Class reweighting or resampling also helps. Trap: Defaulting to accuracy or a fixed 0.5 threshold, or trusting ROC-AUC under extreme imbalance — ROC-AUC can look deceptively high while the model is nearly useless on the minority class.
How do you extend logistic regression to multi-class problems?
Two approaches: one-vs-rest (OvR) trains one binary classifier per class and picks the highest score; multinomial (softmax) logistic regression replaces the sigmoid with a softmax over K classes and optimizes a single cross-entropy objective jointly. Trap: Saying logistic regression is binary-only. Softmax logistic regression handles K classes natively and is exactly what the output layer of a neural network with no hidden layers computes.
11
🎯One recipe: pick a distribution for y, then point its natural parameter straight at θᵀx.
One template, infinite models: pick a distribution for your output, point its "natural knob" straight at θᵀx, and read off the mean — that is all a GLM is. Gaussian knob gives linear regression, Bernoulli knob gives logistic regression, Poisson knob gives count regression, all from the same fitting recipe.
THE EXPONENTIAL FAMILY — ONE FORM TO RULE THEM ALL
Nearly every distribution you know lives inside this single shell:
$$p(y;\eta)=b(y)\exp\!\bigl(\eta^{\top}T(y)-a(\eta)\bigr)$$base measure × e raised to (natural parameter · sufficient statistic − normalizer)
η (eta)natural parameter — the single knob that tunes the distribution
T(y)sufficient statistic — all the data information that matters (usually just y)
a(η)log-partition — ensures the distribution sums to 1; its derivatives give moments for free
b(y)base measure — η-independent factor; shapes the support
The magic: a(η) alone encodes the whole distribution. Take its derivative and you get the mean; take the second derivative and you get the variance.
FREE MOMENTS FROM THE NORMALIZER
$$\mathbb{E}[y;\eta]=a'(\eta),\qquad \operatorname{Var}[y;\eta]=a''(\eta)$$first derivative of log-partition = mean; second derivative = variance
Because a(η) is convex, GLM fitting is always a convex optimization problem — a single global minimum, no local traps. This is why the same gradient descent / Newton machinery works for every member of the family.
THE GLM RECIPE — THREE INGREDIENTS
- Pick a family. Gaussian for continuous, Bernoulli for binary, Poisson for counts.
- Wire the natural parameter. Set η = θᵀx (linearity lives here, not in the mean).
- Predict the mean. Output ŷ = 𝔼[y|x] = a'(η) = g⁻¹(θᵀx).
| Distribution | Link g | Response g⁻¹ | Model |
|---|---|---|---|
| Gaussian | identity | θᵀx | Linear regression |
| Bernoulli | logit | σ(θᵀx) | Logistic regression |
| Poisson | log | exp(θᵀx) | Poisson regression |
The MLE loss drops out automatically: squared error for Gaussian, cross-entropy for Bernoulli, Poisson deviance for Poisson — no separate choice needed.
LOGISTIC REGRESSION — THE GLM POSTER CHILD
Bernoulli with η = log(p/(1−p)) the log-odds. Set η = θᵀx, invert: p = σ(θᵀx). The model is linear in the log-odds (logit), not in the probability. One-unit change in x_j shifts the log-odds by θ_j; exp(θ_j) is the odds ratio. No closed-form solution — use gradient descent or Newton's method (IRLS). Decision boundary is the hyperplane θᵀx = 0.
⚠ Clears up — "logistic regression is nonlinear" The sigmoid makes the probability nonlinear in the features, but logistic regression is perfectly linear in the log-odds. That is why it is still called a linear model and fits inside the GLM framework.
◆ Interview probe "Can you use MSE as the loss for logistic regression?" → You can, but you shouldn't: MSE composed with the sigmoid is non-convex (multiple local minima, vanishing gradients on confident wrong predictions). Log-loss is the principled MLE objective and is convex in the weights.
Remember GLMs are one recipe — pick a distribution, wire η = θᵀx, read the mean from a'(η) — and the right loss function follows for free.
Tricky interview questions 10
What is the core difference between linear and logistic regression?
Linear regression predicts a continuous output as a linear function of features, fit by minimizing squared error (MLE under Gaussian noise). Logistic regression predicts class probability by passing the same linear combination through a sigmoid, and is fit by maximizing Bernoulli likelihood (cross-entropy loss). Both are GLMs — they differ only in the chosen distribution. Trap: Saying "logistic is for classification, linear for regression" and stopping. The deeper answer: logistic regression is linear in the log-odds, not in the probability.
Why can't we just use linear regression for binary classification?
Linear regression gives unbounded outputs that can't be read as probabilities, and assumes homoscedastic Gaussian errors — both violated by 0/1 targets. Extreme points also pull the fitted line and shift the decision boundary arbitrarily. Trap: Thinking the only problem is output range. Even after clipping to [0,1], the constant-variance and error-normality assumptions are broken, and leverage points distort the boundary.
Why does logistic regression use log-loss instead of MSE?
Log-loss is the exact negative log-likelihood of the Bernoulli model — the principled MLE objective. It is convex in the weights (unique global optimum, well-behaved gradients). MSE composed with the sigmoid is non-convex and produces vanishing gradients when the model is confidently wrong. Trap: Saying MSE "can't be optimized." It can; the real issues are non-convexity, poor gradients, and that log-loss — not MSE — is what MLE demands.
How are logistic regression parameters estimated? Is there a closed-form solution?
By maximum likelihood, minimizing cross-entropy loss. Unlike linear regression's normal equation, there is no closed-form solution, so logistic regression is solved iteratively via gradient descent or Newton's method (IRLS — iteratively reweighted least squares). Trap: Claiming logistic regression has a closed-form solution like OLS. It does not — the sigmoid's nonlinearity prevents analytic inversion.
How do you interpret a logistic regression coefficient?
A one-unit increase in feature x_j increases the log-odds of the positive class by θ_j, holding other features fixed. exp(θ_j) is the odds ratio — a factor by which the odds multiply. For example, θ_j = 0.7 means the odds roughly double (e^0.7 ≈ 2). Trap: Saying the coefficient changes the probability by a fixed amount. The effect on probability is nonlinear and depends on the baseline; the coefficient directly scales the log-odds, not the probability.
What are the assumptions of linear regression, and which ones affect predictions vs inference?
Linearity, independence of errors, normally distributed errors, and homoscedasticity (constant variance), plus no severe multicollinearity. Normality and homoscedasticity mainly affect the validity of confidence intervals and p-values (inference), not just point predictions. Trap: Saying "the features must be normally distributed." It is the residuals that are assumed normal — not the predictors or the marginal response.
What is the decision boundary of logistic regression? When does it fail?
The decision boundary is the hyperplane θᵀx = 0 (where predicted probability = 0.5) — it is linear in the input features. Logistic regression fails when classes are not approximately linearly separable; adding polynomial or interaction features can create nonlinear boundaries in feature space. Trap: Assuming logistic regression can fit any decision boundary. Without feature engineering, it is strictly linear.
Why does L1 regularization produce sparse weights while L2 does not?
L1 adds λΣ|w| to the loss; its constraint region is a diamond with corners on the axes, so the optimal solution often lands exactly at a corner where most weights are zero. L2's circular constraint shrinks all weights smoothly toward zero but never exactly reaches it. Trap: Saying L2 also zeroes out coefficients, or that L1 is always better. Only L1 yields exact zeros; L2 handles correlated features more gracefully and is differentiable everywhere.
What is the normal equation, and when would you prefer gradient descent over it?
The normal equation w = (XᵀX)⁻¹Xᵀy solves linear regression exactly in one step. Forming XᵀX is O(nd²) and inverting it is O(d³), so for large d (many features) or when XᵀX is near-singular (multicollinearity, d > n), gradient descent is faster and more numerically stable. Trap: Assuming the normal equation always wins. It scales cubically in features and fails when XᵀX is singular — common with high-dimensional or correlated predictors.
What is the GLM recipe, and how do linear and logistic regression both fit inside it?
A GLM picks an exponential-family distribution for y, sets the natural parameter η = θᵀx, and predicts the mean as a'(η). Gaussian distribution with identity link recovers linear regression (a'(η) = η = θᵀx). Bernoulli distribution with logit link recovers logistic regression (a'(η) = σ(θᵀx)). The MLE loss drops out automatically from the chosen distribution. Trap: Thinking linear and logistic regression are fundamentally different algorithms. They are the same GLM recipe with different noise distributions — same fitting machinery, different response functions.
12
🎯Discriminative draws the border; generative paints both countries and asks which one you came from.
Discriminative models draw the border between countries; generative models paint each country's full landscape, then ask "which country does this point belong to?" — using Bayes' rule to flip from "what does class k look like?" to "which class fits this point?"
THE BAYES FLIP
Instead of learning $p(y\mid x)$ directly, a generative model learns the class-conditional density $p(x\mid y)$ and the class prior $p(y)$, then applies Bayes' rule:
$$\hat{y} = \arg\max_y\; p(x\mid y)\,p(y)$$pick the class whose "picture" best matches x, weighted by how common that class is
The denominator $p(x)$ is the same for every class, so it drops out when you just need the argmax.
Discriminativemodels $p(y\mid x)$ directly — logistic regression, SVMs, neural nets
Generativemodels $p(x\mid y)$ and $p(y)$ — GDA, Naive Bayes, VAEs, GANs, diffusion
Bonusgenerative also gives $p(x)$: detect outliers, generate new samples, semi-supervised learning
GAUSSIAN DISCRIMINANT ANALYSIS (GDA)
Fit a Gaussian to each class — same shape $\Sigma$, different center $\mu_k$. No gradient descent; all closed-form MLE:
$$\hat\mu_k = \text{mean of class-}k\text{ points},\quad \hat\Sigma = \text{pooled within-class covariance}$$class centers from averages; shared spread from deviations around each center
Because $\Sigma$ is shared, the Bayes decision boundary is linear (LDA). Give each class its own $\Sigma_k$ and the boundary goes quadratic (QDA).
NAIVE BAYES — SCALE FOR FREE
For discrete or high-dimensional features (text, pixels), assume features are conditionally independent given the class:
$$p(x\mid y) = \prod_{j=1}^{d} p(x_j\mid y)$$the joint likelihood is just a product of one-dimensional look-ups
This is almost always wrong — words and pixels are correlated — but surprisingly often right enough for classification, because you only need the correct argmax, not calibrated probabilities. And it fits in $O(d)$ parameters instead of $O(d^2)$.
| Naive Bayes (generative) | Logistic Regression (discriminative) | |
|---|---|---|
| Assumption | Strong (conditional independence) | Weaker (linear log-odds) |
| Small data | Wins — assumption acts like a prior | High variance, can overfit |
| Large data | Hits asymptotic ceiling | Lower asymptotic error |
| Missing features | Easy — just drop the factor | Needs imputation |
⚠ Clears up — "GDA is discriminative, right?" Despite the word "discriminant" in its name, GDA is a generative model. It models $p(x\mid y)$ and $p(y)$, then uses Bayes to classify. Logistic regression — which learns $p(y\mid x)$ directly — is the discriminative counterpart.
◆ Interview probe Q: Naive Bayes and logistic regression are a generative–discriminative pair. When would you choose each? → With limited labeled data, Naive Bayes often wins because its strong independence assumption acts like a built-in prior, reducing variance. As data grows, logistic regression's lower asymptotic error takes over. Naive Bayes also handles missing features naturally and requires no gradient descent.
Remember Generative models paint both classes and Bayes-flip to classify; the strong assumptions hurt asymptotically but save you when data is scarce.
Tricky interview questions 10
What is the real difference between a generative model and a discriminative model?
A generative model learns the joint distribution $P(x,y)$ via $P(x\mid y)$ and $P(y)$, so it can also synthesize new $x$. A discriminative model directly learns $P(y\mid x)$ without ever modeling how $x$ is distributed. Trap: Saying generative models "generate images/text" and discriminative ones "classify." The real line is which distribution is modeled — Naive Bayes is generative but used purely for classification.
Naive Bayes and logistic regression are a generative–discriminative pair. Which wins with little data vs. lots, and why?
Naive Bayes (generative) converges to its (higher) asymptotic error very fast, so it often dominates with small datasets; its strong assumptions act like a prior that reduces variance. Logistic regression has lower asymptotic error and overtakes Naive Bayes as data grows. Trap: Assuming discriminative is always better — it only wins with enough data.
Why is Naive Bayes called "naive," and why does it work even when the assumption is violated?
It assumes all features are conditionally independent given the class label, which almost never holds. It works because correct classification only requires the right argmax of the posterior — probabilities don't need to be well-calibrated, just ranked correctly, and the independence error often cancels across classes. Trap: Concluding violated independence means poor accuracy — calibration suffers more than rank ordering.
In GDA, why does a shared covariance matrix give a linear boundary, while separate covariances give quadratic?
Taking $\log p(y\mid x) \propto \log p(x\mid y) + \log p(y)$ with equal $\Sigma$ causes the quadratic terms in $x^\top \Sigma^{-1} x$ to cancel, leaving a linear function of $x$. With per-class $\Sigma_k$ the quadratic terms do not cancel, giving a quadratic boundary. Trap: Assuming LDA is always better — QDA fits more expressively but needs more data to estimate each $\Sigma_k$.
What advantages do generative classifiers have over discriminative ones beyond raw accuracy?
Generative models naturally handle missing features (drop that factor), enable outlier/anomaly detection via $p(x)$, support semi-supervised learning with unlabeled data, and can generate new samples. Trap: Forgetting missing-data handling — discriminative models need imputation, while generative models just marginalize over missing features.
GDA assumes class-conditional Gaussians with shared covariance. What breaks if you use QDA on a tiny dataset?
QDA estimates a separate $d \times d$ covariance matrix per class, requiring $O(d^2)$ parameters each. With few samples those estimates become noisy and singular, leading to poor generalization. Shared-$\Sigma$ LDA uses far fewer parameters and generalizes better when $n$ is small relative to $d$. Trap: Saying QDA is "more accurate" in general — it only wins when data is plentiful enough to estimate each $\Sigma_k$ reliably.
Explain the reparameterization trick in VAEs. Why is it necessary?
Instead of sampling $z \sim \mathcal{N}(\mu, \sigma^2)$ directly (a non-differentiable node), write $z = \mu + \sigma \cdot \varepsilon$ with $\varepsilon \sim \mathcal{N}(0,1)$. Randomness moves to a fixed input, making the path to $\mu$ and $\sigma$ fully differentiable so backpropagation trains the encoder. Trap: Treating it as numerical convenience — without it, gradients cannot flow into encoder parameters and you'd need high-variance estimators like REINFORCE.
What is posterior collapse in a VAE, and what causes it?
Posterior collapse is when the encoder's approximate posterior $q(z\mid x)$ collapses to the prior, making the latent code $z$ unused — the decoder learns to reconstruct without $z$. It is caused by a too-powerful decoder (especially autoregressive) overpowering the KL term early in training. Trap: Thinking more KL weight always helps — over-weighting KL aggressively can itself degrade reconstruction. $\beta$-VAE deliberately tunes this trade-off.
Why does a GAN's discriminator cause vanishing gradients, and what does Wasserstein loss fix?
When the discriminator becomes too strong, JS divergence saturates and gives near-zero gradients to the generator. WGAN replaces JS divergence with the Wasserstein (Earth-Mover) distance, which provides smooth, non-vanishing gradients even when real and fake distributions barely overlap, stabilizing training. Trap: Saying the WGAN critic outputs a probability — it must be 1-Lipschitz and outputs an unbounded real-valued score, not a value in $[0,1]$.
Compare GANs, VAEs, and diffusion models on sample quality, diversity, training stability, and inference speed.
GANs: sharp samples, fast single-pass inference, but unstable training and prone to mode collapse. VAEs: stable, interpretable latent space, but blurrier samples. Diffusion: state-of-the-art quality and diversity, stable training, but slow iterative sampling (100–1000 steps) unless distilled. Trap: Calling diffusion strictly best — its main cost is slow sampling, which distillation and fast ODE solvers partially address.
13
🎯k-NN doesn’t train — it just asks the neighbors. Small k = nervous, large k = lazy.
KNN is the ultimate procrastinator: it skips training entirely, memorizes all the data, and only does any work when you ask for a prediction — then it sprints to your neighbors and takes a vote.
THE CORE IDEA: ASK YOUR NEIGHBORS
To classify a new point, find the k closest points in the training set (by distance), then take a majority vote (classification) or average (regression). No model is built. No weights are learned. The data is the model.
Classificationmajority vote among k neighbors
Regressionmean (or weighted mean) of k neighbors' values
Distance metricsEuclidean (default), Manhattan (L1), Cosine (text/embeddings), Hamming (categorical)
k IS YOUR BIAS–VARIANCE DIAL
Small k: wiggly boundary, low bias, high variance — k=1 memorizes every training point perfectly. Large k: smooth boundary, high bias, low variance — at k=N you just predict the global majority every time. Cross-validate to find the sweet spot; pick odd k to break ties.
$$\hat{y} = \text{majority}\{y_i : i \in \mathcal{N}_k(x)\}$$predict by vote among the k nearest neighbors of x
TWO THINGS THAT WILL BREAK IT
| Problem | Why it hurts | Fix |
|---|---|---|
| Unscaled features | Income (0–200k) swamps age (0–100) — distance is dominated by the big-range feature | Z-score or min-max scale everything; fit scaler on train only |
| High dimensions | Distances concentrate — nearest and farthest neighbors become almost equal; "close" stops meaning anything | PCA/embedding first; use ANN (HNSW, FAISS) instead of exact search |
COST STRUCTURE: LAZY ≠ CHEAP
"No training" just moves the cost to serving time. Every prediction scans all N training points in O(N·d) — painful at scale. Speed-ups: KD-tree / ball-tree for low-to-moderate dimensions; ANN indices (HNSW, LSH, FAISS) for high dimensions. KD-trees degrade back to brute-force past ~20 dims — do not recommend them for embeddings.
⚠ Clears up — KNN vs K-Means Both use a "K" and distances, but they solve different problems. KNN is supervised prediction using K neighbors per query; K-means is unsupervised clustering that finds K centroids by iterating. Completely different algorithms — conflating them is a classic interview stumble.
◆ Interview probe "Why can't you just pick a larger K to be safer?" → Because large K oversmooths. You push toward the global majority class and lose all local structure. At K=N on a 90/10 imbalanced dataset, you predict the majority class 100% of the time. It's a bias–variance tradeoff, not a monotonic improvement.
Remember KNN defers everything to inference — the cost is in serving, not training — and it fails silently without feature scaling and without checking whether distances still mean something in your feature space.
Tricky interview questions 10
KNN is called a "lazy learner." What does that actually imply about training vs. inference cost?
Training is nearly free (just store the data), but all computation is deferred to query time — each prediction scans N training points, making inference O(N·d). "No training" does not mean cheap overall; the cost is shifted, not eliminated. Trap: Assuming lazy = fast overall. Inference becomes the bottleneck at scale.
What happens to KNN at the two extremes: k=1 and k=N?
k=1 memorizes the training set perfectly (zero training error) but is wildly sensitive to noise — high variance. k=N ignores all local structure and predicts the global majority class every time — maximum bias. Neither generalizes well; cross-validate to find the middle. Trap: Thinking larger k is always safer — too large just means underfitting.
Why is feature scaling mandatory for KNN, and what scaling mistake can leak information?
Euclidean distance is scale-dependent, so a feature with a large range (e.g. income in thousands) dominates the distance and drowns out small-range features. The leakage trap: fitting the scaler on the full dataset (train + test) — you must fit on train only, then transform test.
Explain the curse of dimensionality and why it specifically cripples KNN compared to other algorithms.
As dimensions grow, data becomes exponentially sparse and all pairwise distances converge — nearest and farthest neighbors become nearly equidistant, so "nearest" loses discriminative meaning. KNN is entirely distance-dependent, so this kills it; parametric models learn a compact decision boundary and don't suffer in the same way. Trap: Framing it only as a speed problem — the deeper issue is that distances stop being meaningful.
When would you use cosine distance instead of Euclidean for KNN?
When the direction of a vector matters more than its magnitude — text TF-IDF vectors, word/sentence embeddings, and other high-dimensional sparse representations. A short document and a long one on the same topic should be close; Euclidean distance would penalize the magnitude difference. Trap: Defaulting to Euclidean for all continuous features; embeddings call for cosine.
How does KNN do regression, and can it extrapolate beyond the training range?
It predicts the (optionally distance-weighted) average of the k nearest neighbors' target values instead of voting. It cannot extrapolate — it can only average stored values, so predictions are always bounded within the training target range. Trap: Assuming it behaves like a linear model that can project outside the data.
What is distance-weighted KNN and when does it genuinely help?
Neighbors vote with weight proportional to 1/distance (or a kernel weight) so closer points count more. It reduces sensitivity to the exact choice of k, naturally breaks ties, and helps when the boundary between classes is sharp — far-away neighbors in the wrong class matter less. Trap: Dismissing it as cosmetic; weighting can meaningfully improve accuracy and lets you use a larger, more stable k.
Why does KNN struggle on imbalanced datasets, and what's the wrong fix?
Majority-vote neighborhoods are populated mostly by the abundant class, so the minority class is predicted rarely. Correct fixes: distance weighting, SMOTE/resampling, or class-weighted votes. Trap: Raising k to "stabilize" predictions — it makes minority recall even worse by pulling in more majority-class neighbors.
You have 10 million training points and need KNN at low latency. What's your approach?
Exact brute-force is O(N·d) per query — too slow. For moderate dimensions use KD-tree or ball-tree; for high-dimensional embeddings use approximate nearest neighbor (ANN) methods like HNSW (Hierarchical Navigable Small World) or FAISS. ANN trades a small accuracy loss for orders-of-magnitude speedup. Trap: Recommending KD-trees for high-dimensional data — they degrade to brute force past ~20 dims.
How is KNN different from K-Means? (Both use "K" and distances.)
KNN is supervised (labeled data, predicts a class or value per query using k neighbors); K-means is unsupervised (no labels, finds k cluster centroids by iterating). They solve entirely different problems — labeled prediction vs. unlabeled grouping. Trap: Conflating them because both rely on k and distances; this is a common stumble that signals shallow understanding.
14
🎯Pick the widest street between the classes — only the curb-stones (support vectors) hold it up.
Imagine stretching a rubber sheet until the two colored blobs pull apart — then slicing the widest possible gap between them. That is SVMs with kernels: bend the space (kernel trick), then cut the fattest street (max-margin).
THE KERNEL TRICK — BENDING SPACE FOR FREE
A feature map φ lifts each point into a richer space where a linear cut works. The problem: φ(x) can be infinite-dimensional, so you never want to compute it. A kernel sidesteps this entirely.
$$K(x,z)=\phi(x)\cdot\phi(z)$$inner product in the lifted space — computed from the originals, φ never materialized
If your algorithm only touches data through dot products, swap every ⟨xi,xj⟩ for K(xi,xj). You silently work in the lifted space at the cost of one cheap scalar evaluation. The RBF kernel’s Taylor expansion has infinitely many polynomial terms — φ is infinite-dimensional and could never be stored; the kernel is the only way in.
VALID KERNELS — MERCER’S CONDITION
Not every symmetric function is a real inner product. K is valid iff for any finite point set the Gram matrix Kij = K(xi,xj) is symmetric positive semidefinite (vᵀKv ≥ 0 for all v). PSD keeps the dual QP convex — guaranteeing a unique solution.
Linear K = x·zno lift, best for high-dim/sparse (text)
Polynomial K = (1+x·z)dall monomials up to degree d (finite φ)
RBF K = exp(−‖x−z‖² / 2σ²)infinite φ; similarity = 1 at x=z, decays to 0
Building new kernels: sums and products of valid kernels are valid. Differences are not — subtracting can break PSD.
SVM — THE WIDEST STREET
Among all separating hyperplanes, pick the one with the largest gap to the nearest points of either class. Those boundary points are the support vectors — remove any other point and the model is unchanged.
$$\text{margin} = \frac{2}{\|w\|}$$width of the slab; maximizing margin = minimizing ½‖w‖² = L2 regularization
The soft-margin SVM adds slack: pay C × hinge-loss for violations. Large C → penalize violations hard → narrow margin (overfit risk). Small C → tolerate violations → wide margin (more regularized). For RBF, γ = 1/(2σ²) sets each support vector’s “reach”: high γ → wiggly boundary (overfit); low γ → smooth (underfit). C and γ must be tuned jointly — if γ is too large, no value of C can save you.
| Linear SVM | Kernel SVM | |
|---|---|---|
| Training cost | O(n × features) | O(n²)–O(n³) |
| Prediction cost | O(features) | O(SVs × features) |
| Best for | large / sparse / text | low-to-mid-dim nonlinear |
| Model size | fixed (parametric) | grows with SVs (non-parametric) |
WHY DUAL? WHY KERNELIZABLE?
In the dual, training points appear only inside dot products xi·xj, and the predictor is a weighted sum of kernels with support vectors. Swap each dot product for K(xi,xj) and you’re in the lifted space. The primal weight vector w lives explicitly in feature space — for infinite-dimensional φ it cannot be stored, so the primal is kernelized only indirectly (representer theorem).
⚠ Clears up — “RBF always beats linear” On very high-dimensional or sparse data (text classification, n_features ≫ n_samples) a linear SVM usually matches RBF accuracy and is orders of magnitude faster. RBF can actually overfit badly there. Try linear first.
◆ Interview probe If your RBF-SVM is overfitting, is it enough to just lower C? → No. If γ is too large, each support vector’s influence barely extends past itself and C cannot regularize that away. You must lower γ and tune C jointly — grid-search or randomized search over both simultaneously.
Remember The kernel trick evaluates an infinite-dimensional dot product as one cheap scalar — so SVMs can cut the widest margin in a space they never explicitly enter.
Tricky interview questions 10
What is the kernel trick, and what does it save you from computing?
A kernel K(x,z) = φ(x)·φ(z) returns the inner product of two points in a high-dimensional lifted space directly from the originals — without ever forming φ(x). For RBF, φ is infinite-dimensional and literally cannot be materialized. Trap: saying the data is "projected into high dimensions." The point is φ is never computed; only the scalar kernel value is.
Why can you apply the kernel trick to the dual SVM but not straightforwardly to the primal?
In the dual, training points appear only inside dot products xi·xj, and the predictor is a weighted sum of kernel evaluations with support vectors — swap each dot product for K and you’re done. The primal depends on an explicit weight vector w in feature space, which is infinite-dimensional for RBF and cannot be stored. Trap: knowing "use the dual" without saying why: data enters solely through dot products, making the substitution exact.
What makes a function a valid kernel (Mercer’s condition)?
A symmetric function K is a valid kernel iff for any finite set of points the Gram matrix Kij = K(xi,xj) is symmetric positive semidefinite. PSD-ness guarantees an implicit feature map φ exists with K = φ·φ, and keeps the dual QP convex. Trap: answering "it must be symmetric" alone — symmetry is necessary but not sufficient; PSD is the real requirement.
What does the γ parameter in the RBF kernel control, and which direction causes overfitting?
γ sets the inverse radius of influence of each support vector. High γ = narrow reach = wiggly, complex boundary (overfit). Low γ = wide reach = smooth boundary (underfit). Trap: reversing the direction — large γ is more complex, not smoother. Also, γ is a kernel parameter independent of the regularization parameter C.
If your RBF-SVM badly overfits, can you fix it by lowering C alone?
Not reliably. If γ is too large, each support vector’s influence shrinks to essentially itself, and no value of C can compensate — you must also reduce γ. C and γ have to be tuned jointly (grid or random search). Trap: assuming C alone governs overfitting. When γ is very large, C becomes nearly irrelevant — this is the classic scikit-learn gotcha.
What are support vectors, and why is the SVM’s solution called sparse?
Support vectors are training points with nonzero Lagrange multipliers — those on the margin, inside it, or misclassified. The decision boundary depends only on them; all other points have αi = 0 and can be removed without changing the model. Trap: saying support vectors are only "the points closest to the boundary." Points that violate the margin are also support vectors.
Why does maximizing the SVM margin act as L2 regularization?
The geometric margin equals 2/‖w‖, so maximizing it is equivalent to minimizing ½‖w‖² — exactly an L2 penalty. A larger margin tolerates more perturbation before misclassifying, giving better generalization bounds. Trap: just saying "it’s more robust" without the ‖w‖ link: margin = 2/‖w‖ so maximizing margin ⇔ minimizing ½‖w‖².
What is hinge loss and how does it differ from logistic loss?
Hinge loss = max(0, 1 − yi(w·xi+b)): zero for correctly-classified points beyond the margin, linear penalty otherwise. Logistic loss is never exactly zero. Hinge’s exact zeros create sparsity (support vectors); logistic regression uses all training points and produces calibrated probabilities. Trap: confusing them — hinge creates the sparse SVM solution; logistic loss does not.
When would you choose a linear kernel over RBF?
For high-dimensional or sparse data (text, n_features ≫ n_samples): linear SVM trains in O(n × features) vs. O(n²)–O(n³) for kernel SVM, and usually matches RBF accuracy there. RBF can overfit on high-dimensional data. Rule of thumb: try linear first, add RBF only if the boundary is genuinely nonlinear. Trap: defaulting to RBF for everything — on large/sparse datasets it is far slower and can be worse.
How do you build new valid kernels from existing ones?
Valid (PSD) kernels are closed under nonneg scaling (cK, c≥0), sum (K1+K2), product (K1·K2), and composition with a power series with nonneg coefficients — e.g., exp(K). For example, (1+x·z)d is valid because it’s built from sums and products of the linear kernel. Trap: thinking K1−K2 is valid — subtraction can break PSD, so differences are generally not kernels.
15
🎯SGD stumbles downhill, momentum gives it legs, Adam gives it GPS; Newton reads the map (but it’s pricey).
Optimization is the engine under every ML model — it walks downhill on the loss surface, one gradient step at a time. SGD stumbles, momentum gives it legs, Adam gives it GPS, Newton reads the full map but charges a fortune.
CONVEXITY — THE GUARANTEE
A loss surface is convex if it looks like a single bowl: no false valleys, no hidden traps. The payoff is huge — every local minimum is the global minimum, so gradient descent is guaranteed to find the best answer regardless of where you start. Linear regression, logistic regression, and SVMs all live here.
Convex testHessian ∇²f is positive semidefinite everywhere
Deep netsNon-convex — local minima and saddle points everywhere, no guarantees
Practical upshotHigh-dim non-convex surfaces have mostly equal-quality minima; saddle points are the real enemy
GRADIENT DESCENT & THE LEARNING RATE
The gradient points uphill; step the opposite way. The learning rate α controls how far.
$$\theta \;:=\; \theta - \alpha\,\nabla_\theta J(\theta)$$nudge θ downhill by step size α times the gradient
α too largeovershoots the valley — loss oscillates or diverges
α too smallcrawls — stable but painfully slow
Mini-batch SGDstandard in practice: cheap steps, built-in noise that escapes saddles & flat minima
SGD noise is not purely harmful — it acts as implicit regularization, biasing toward flat minima that generalize better. Large batches with no learning-rate rescaling often generalize worse.
MOMENTUM → RMSPROP → ADAM
Momentum accumulates a velocity of past gradients, damping zig-zags in ravines and accelerating along consistent directions. RMSprop tracks a moving average of squared gradients to give each parameter its own adaptive step size. Adam = momentum + RMSprop + bias correction — the default for most deep learning.
$$\hat{m}_t / (\sqrt{\hat{v}_t} + \varepsilon)$$Adam effective step: bias-corrected first moment divided by root of bias-corrected second moment
Bias correction matters early in training because both moment estimates start at zero and would otherwise be too small. It fades as t grows.
| Method | What it adds | Weakness |
|---|---|---|
| SGD | baseline | zig-zags, slow ravines |
| SGD + Momentum | velocity smoothing | single global lr |
| AdaGrad | per-param lr | lr decays to zero, stalls |
| RMSprop | fixed-window per-param lr | no momentum |
| Adam | momentum + per-param lr + bias fix | can generalize slightly worse than tuned SGD |
SECOND-ORDER METHODS & SCHEDULES
Newton's method uses the Hessian for curvature-aware steps — no manual learning rate, fast near the minimum. But inverting an n×n Hessian costs O(n³), which is completely infeasible for millions of parameters. We use first-order SGD/Adam with cheap approximations (L-BFGS, K-FAC) instead.
A learning-rate schedule (warmup + cosine/step decay) lets the optimizer take bold steps early and settle precisely late — a constant lr keeps parameters bouncing around the minimum forever.
⚠ Clears up — Adam removes the need to tune the learning rate Adam adapts a per-parameter scale factor via the second moment, but the base learning rate is still a critical hyperparameter. You still need to tune and schedule it — usually with warmup and cosine decay.
◆ Interview probe "Why might well-tuned SGD+momentum beat Adam on image classification?" → Adam drives training loss down faster, but often finds sharper minima that generalize slightly worse. Well-tuned SGD+momentum tends to find flatter minima, producing higher test accuracy on CNN vision benchmarks — faster optimization does not equal better generalization.
Remember Convexity guarantees descent finds the global minimum; for non-convex deep nets, SGD noise escapes saddles, Adam adapts per-parameter step sizes, and the learning rate schedule is still the knob that determines whether you settle or bounce.
Tricky interview questions 10
Explain gradient descent. What does the update rule look like and what is the learning rate?
Gradient descent iteratively moves parameters opposite the gradient: θ := θ − α ∇J(θ). The gradient points in the direction of steepest increase; negating it descends the loss. The learning rate α sets the step size. Trap: saying GD finds the global minimum. It follows the local negative gradient and converges to a stationary point (local min or saddle) — only convex losses guarantee the global optimum.
What is the difference between batch, mini-batch, and stochastic gradient descent? Which do you use in practice?
Batch GD uses the full dataset per step (exact gradient, expensive). True SGD uses one example (cheap, high variance). Mini-batch uses a small batch (32–512), balancing variance and compute — it's the real default in deep learning because it vectorizes well on GPUs. Trap: "SGD" in most deep-learning code means mini-batch, not single-sample updates.
Why and how does SGD noise help optimization? Is the noise purely harmful?
Stochastic noise helps the optimizer escape saddle points and sharp local minima, and acts as implicit regularization biasing toward flatter minima that generalize better. Very large batches that reduce noise to near-zero often generalize worse — the "large-batch generalization gap." Trap: treating noise as strictly harmful. It's often a feature, not a bug.
How does Adam work, and how does it combine momentum and RMSprop?
Adam maintains an EMA of the gradient (first moment, like momentum) and of the squared gradient (second moment, like RMSprop), bias-corrects both, then updates with m̂ / (√v̂ + ε). This provides momentum's directional smoothing plus per-parameter adaptive step sizes. Trap: saying Adam adapts the learning rate globally — it adapts a per-parameter scale factor; the base lr is still a hyperparameter you must set and schedule.
Why does Adam need bias correction on its moment estimates?
Both moment EMAs are initialized to zero, so early in training they are biased toward zero, producing under-scaled updates. Dividing by (1 − β¹&lsup;t) and (1 − β²&lsup;t) corrects this; the factor approaches 1 as t grows, so the correction matters chiefly in the first few dozen steps. Trap: saying bias correction matters throughout training — it only significantly affects early updates.
Adam often trains faster — so why do people still use SGD with momentum, especially for vision models?
Adam usually drives training loss down faster but often generalizes slightly worse; well-tuned SGD+momentum tends to find flatter minima that generalize better, so it frequently reaches higher test accuracy on CNN image models. Trap: "Adam is always better." Faster training-loss descent does not imply better test performance.
What is convexity and why does it matter for optimization?
A function is convex if the chord between any two points lies on or above the curve (equivalently, a positive semidefinite Hessian). It matters because every local minimum is a global minimum — gradient methods provably reach the best answer. Trap: "convex means one unique minimum." Uniqueness requires strict convexity; a flat-bottomed convex function can have infinitely many minimizers.
Why are saddle points often considered a bigger problem than local minima in high-dimensional deep learning?
At a saddle the gradient is zero, but curvature is positive in some directions and negative in others. In high dimensions, critical points are overwhelmingly saddles (not true local minima), and gradient descent stalls on the flat plateau around them far more often than at bad local minima. Trap: thinking momentum/SGD cannot escape saddles — SGD noise and momentum's drift do help; plain GD lingers the longest.
What are vanishing and exploding gradients, and how do you address them?
Chaining many layer Jacobians through backprop makes gradients shrink to zero (vanishing, common with saturating sigmoid/tanh) or blow up (exploding). Fixes: ReLU activations, Xavier/He initialization, batch/layer normalization, residual connections, and gradient clipping for exploding gradients. Trap: using gradient clipping to fix vanishing gradients — clipping only caps exploding gradients; vanishing needs architectural and activation fixes.
Compare AdaGrad, RMSprop, and Adam. When would AdaGrad fail?
AdaGrad accumulates all past squared gradients, so its per-parameter effective learning rate decays monotonically and eventually stalls learning — bad for long deep-net training. RMSprop fixes this by using an EMA of squared gradients (a fixed effective window). Adam adds momentum's first moment and bias correction on top of RMSprop. Trap: recommending AdaGrad for deep net training — its ever-growing denominator drives effective lr toward zero.
16
🎯The gap between training error and truth shrinks like √(capacity / data).
Every model lives on a see-saw: make it richer and it stops missing patterns (bias drops) but starts memorising noise (variance rises). Your job is to find the sweet spot before the see-saw tips.
THE THREE-TERM SPLIT
Squared prediction error on a fresh point is not one thing — it is three:
$$\mathbb{E}[(y-\hat{f})^2] = \text{Bias}[\hat{f}]^2 + \text{Var}[\hat{f}] + \sigma^2$$average error = systematic miss² + prediction wobble + irreducible noise
Bias²how far your average prediction is from truth — a flaw in the model's shape
Variancehow much the prediction wiggles across different training sets — a flaw in stability
σ²noise baked into y itself; no model ever beats this floor
COMPLEXITY AS THE DIAL
Turn complexity up: training error falls monotonically. Test error is U-shaped — it falls while bias is being fixed, bottoms out, then rises as variance takes over. The bottom of the U is the target. Regularisation (L1, L2, dropout, early stopping) pulls the model toward that bottom by trading a little bias for a lot of variance.
| High Bias (underfit) | High Variance (overfit) | |
|---|---|---|
| Training error | High | Low |
| Validation error | High (≈ training) | High (≫ training) |
| Fix with | More complexity, features, boosting | Regularisation, data, bagging |
TWO FIXES, TWO TARGETS
Variance cures: more data, L1/L2 penalty, dropout, early stopping, bagging (parallel averaging). Bias cures: richer model, more/better features, less regularisation, boosting (sequential error-correction). The diagnostic is a learning curve: if train and val errors are both high and close together, add capacity; if they diverge, add data or regularise.
⚠ Clears up — k-NN direction People think large k = more complex. It is the opposite. k = 1 memorises every point (low bias, high variance); k = N averages everything (high bias, low variance). Larger k is simpler.
◆ Interview probe "Does collecting more training data reduce bias?" → No. More data shrinks variance (the train/val gap), not bias. If training error itself is already high, the model is structurally too simple and more examples will not help. You need a more expressive model.
Remember Bias is a broken compass; variance is a shaky hand — more data steadies the hand, but only a better compass fixes the direction.
Tricky interview questions 10
Explain the bias-variance tradeoff and how it connects to underfitting and overfitting.
Bias is systematic error from a model too simple to capture the true pattern — it underfits. Variance is instability: the model is so sensitive to training data that it fits noise — it overfits. Raising complexity lowers bias and raises variance, so you tune to the sweet spot where total test error is minimised. Trap: claiming you can minimise both simultaneously with a fixed model class; reducing one typically increases the other.
Write the bias-variance decomposition of expected squared error. What are the three terms, and which one can never be eliminated?
E[(y − f̂)²] = Bias[f̂]² + Var[f̂] + σ², where σ² is the irreducible noise in y. Bias² captures how wrong the average prediction is, Var captures how much predictions wobble across training sets, and σ² is a floor no model can beat. Trap: forgetting σ² and thinking perfect training data would allow zero test error.
How do you diagnose high bias vs. high variance from training and validation error curves?
High bias: both training and validation error are high and close together (the model is wrong everywhere). High variance: training error is low but validation error is much higher — a large gap. Trap: looking only at validation error; you must compare it to training error and to the irreducible baseline to name the disease.
Your model overfits. List four concrete fixes and explain why each works.
Collect more data (stabilises estimates, reduces variance without harming bias); add L2/L1 regularisation (shrinks weights, increasing bias slightly but cutting variance substantially); reduce model complexity (fewer parameters = less flexibility to fit noise); use dropout or early stopping (implicit regularisers that prevent full memorisation). Trap: adding more features or a larger model — that worsens overfitting.
Does adding more training data fix a high-bias problem? Justify your answer.
No. More data reduces variance — it narrows the gap between training and validation error. If training error is itself high (high bias), the model's functional form is too simple to capture the pattern, and no amount of new examples changes that. You need a more expressive model or better features. Trap: treating more data as a universal cure; it only cures variance.
How do L1 and L2 regularisation each affect the bias-variance tradeoff, and how do they differ?
Both increase bias slightly and reduce variance by shrinking coefficients. L1 (Lasso) penalises absolute values and pushes some weights exactly to zero, performing feature selection. L2 (Ridge) penalises squared values and shrinks all weights smoothly toward zero, keeping all features active. Trap: saying L2 produces sparse solutions — only L1 zeros out coefficients.
In k-NN, which direction does increasing k move you on the bias-variance curve?
Larger k averages more neighbours, smoothing the decision boundary: bias rises, variance falls. Small k (k = 1) memorises every training point — low bias, high variance. So larger k = simpler model, not more complex. Trap: thinking large k means more complex. The direction is reversed compared to most models.
Why does bagging reduce variance but not bias, while boosting reduces bias but risks increasing variance?
Bagging averages many high-variance models trained on bootstrap samples; averaging reduces variance toward Var/N while bias is unchanged (each tree still has the same structural bias). Boosting trains sequentially, each learner correcting the residual errors of the ensemble — that drives bias down. However, running too many boosting rounds without regularisation can overfit, raising variance. Trap: treating bagging and boosting as interchangeable variance reducers.
Sketch how validation error changes as you increase the regularisation strength λ from 0 to ∞.
At λ = 0 (no regularisation) the model can overfit: low bias, high variance, potentially high validation error. As λ rises, variance falls and validation error typically improves. Beyond the optimal λ, the model is too constrained (high bias) and validation error climbs again. The curve is U-shaped; you pick λ at the bottom via cross-validation. Trap: asserting more regularisation is always better; there is an optimum, not a monotone improvement.
What does a learning curve (error vs. training-set size) tell you, and how do you use it to decide whether to collect more data?
Plot training and validation error as you increase dataset size. If both curves converge at high error, you have high bias — more data will not help; you need a more expressive model. If there is a persistent gap (validation error much higher than training), you have high variance — more data will likely close the gap. Trap: confusing a learning curve (error vs. data size) with a validation curve (error vs. hyperparameter); they answer different questions.
17
🎯L2 shrinks everyone a little; L1 fires the useless ones (sets them to exactly 0).
Regularization is the dial between memorizing and generalizing — L2 shrinks every weight a little, L1 cuts useless ones to exactly zero. Model selection is how you turn that dial honestly, using held-out data your model has never touched.
THE PENALTY DIAL
Every regularizer adds a complexity charge to the loss. Crank λ up and the model gets simpler but less fit; turn it down and it memorizes.
$$\hat\theta = \arg\min_\theta \underbrace{\sum_i L\!\left(y_i, f_\theta(x_i)\right)}_{\text{fit}} + \lambda\,\underbrace{R(\theta)}_{\text{complexity}}$$minimize fit loss plus λ × penalty
L2 / ridgepenalty = ∑θ₂; shrinks all weights, zeroes none; closed form; fixes correlated features
L1 / lassopenalty = ∑|θ|; sets some weights to exactly 0 → built-in feature selection
Elastic netmix of both; α L1 + (1−α) L2; sparsity with ridge stability
GEOMETRY — WHY THE DIAMOND GIVES SPARSITY
Penalized fitting equals "stay inside a budget ball." The loss contours expand outward until they first touch that ball — that contact point is your solution.
L1 = diamond. Corners sit exactly on the axes (one coordinate = 0). The elliptical loss almost always hits a corner first → exact zeros → feature selection.
L2 = sphere. No corners. The contour touches a smooth side, so all weights shrink but none vanish.
HONEST MODEL SELECTION
Training error is rigged — you built the model on those exact points. You need an honest score: one computed on data the model has never influenced.
| Method | When to use | Watch out |
|---|---|---|
| Hold-out split | large data, fast | high variance if data is small |
| k-fold CV (k=5 or 10) | most cases | must fit preprocessors inside each fold |
| LOOCV (k=n) | tiny datasets | high variance, expensive |
| TimeSeriesSplit | time-ordered data | never shuffle; use future-only validation |
Nested CV separates tuning (inner loop) from evaluation (outer loop) so that hyperparameter search does not inflate your reported score.
LEAKAGE — THE SILENT KILLER
Leakage is when information from the validation set bleeds into training. The two flavors:
Feature leakagea feature encodes the answer or uses future data (e.g., "days until default")
Train-test contaminationfitting a scaler or encoder on the full dataset before splitting lets test statistics leak into training
Rule: split first, then fit every transformer. Wrap preprocessing + model in a Pipeline so it refits on each fold automatically.
⚠ Clears up — "CV prevents overfitting" Cross-validation only measures generalization; it does not reduce overfitting. You still need regularization, simpler models, or more data for that. CV just tells you honestly how bad the problem is.
◆ Interview probe You split your data, standardize features, cross-validate, and get 94% accuracy. Your colleague says the pipeline is leaky — how? → If
StandardScaler.fit_transform was called on the full dataset before the CV split, the mean and std of held-out folds leaked into training. Fix: put the scaler inside a Pipeline so it re-fits on each fold's training portion only.Remember L2 shrinks all weights, L1 zeros the useless ones — and no CV score is honest unless every preprocessing step was fitted strictly inside the fold.
Tricky interview questions 10
What is cross-validation and why use it instead of a single train/test split?
Cross-validation rotates a held-out fold through the full dataset and averages the metric, giving a lower-variance, luck-independent estimate of out-of-sample performance — every example serves as validation exactly once. A single split gives you one noisy number that depends heavily on which rows happened to land in each partition. Trap: saying CV "prevents" overfitting. It only measures generalization; it does not change the model's bias or variance.
What is the difference between a validation set and a test set?
The validation set is used repeatedly during development to tune hyperparameters; the test set is touched exactly once at the end to report final performance. Because you optimize against the validation set, the model implicitly fits it, so validation performance is optimistically biased. Trap: treating them as interchangeable — reusing the test set for decisions converts it into a validation set and contaminates your final estimate.
Why does LOOCV have high variance even though it averages n models?
Each of the n models is trained on nearly identical data (n−1 points), so their errors are highly correlated. Averaging correlated quantities does not reduce variance much — you get many fold scores but little independent information. k=5 or k=10 gives a lower-variance estimate at far less compute cost. Trap: assuming more folds always means lower variance; correlation among fold models is the gotcha.
When is plain random k-fold the WRONG choice, and what do you use instead?
Random k-fold is wrong for time series (model trains on the future), grouped data like multiple rows per user (same entity leaks across folds), and class-imbalanced data. Use TimeSeriesSplit, GroupKFold, and StratifiedKFold respectively. Trap: defaulting to random shuffling everywhere — shuffled time-series data produces CV scores that collapse completely in production.
Why is it wrong to fit a scaler before splitting, and how do you fix it?
Fitting a scaler on the full dataset lets the mean and standard deviation of held-out rows leak into training, inflating CV scores optimistically. Fix by splitting first and fitting the transformer only on the training fold, ideally by wrapping everything in a scikit-learn Pipeline so it automatically refits on each fold. Trap: calling fit_transform on all data "just to normalize it" — even unsupervised steps like scaling and PCA leak.
Explain the geometric reason L1 regularization produces sparse solutions but L2 does not.
The L1 constraint region is a diamond with corners on the coordinate axes; loss contours expanding from the unconstrained optimum almost always strike a corner first, landing the solution where at least one weight is exactly zero. The L2 constraint is a smooth sphere with no corners, so the contact point generally has all non-zero coordinates. Trap: saying L2 "can" zero weights — it drives them toward zero but never reaches exactly zero.
What is data leakage, and which form is more subtle and dangerous in practice?
Leakage is when information unavailable at true prediction time influences training, inflating offline metrics that then collapse in production. Target/feature leakage (a feature encodes the answer) is obvious; train-test contamination via preprocessing on the full dataset is subtler and far more common. Trap: thinking leakage only means literally including the label — preprocessing leakage is the silent, widespread form.
If you do feature selection by ranking features on correlation with the target, how must it interact with cross-validation?
Feature selection that looks at the target must happen inside each CV fold using only that fold's training data; selecting once on the full dataset before CV lets the selection see validation rows, biasing the score upward. This routinely yields near-perfect CV scores even on pure noise in high-dimensional, small-n settings. Trap: picking "top features" globally and then cross-validating only the model — a classic, severe leak.
What is nested cross-validation and why does a single CV loop understate the true error when you do extensive hyperparameter search?
A single CV loop finds the best hyperparameter configuration by picking the max over many search evaluations, so the reported score is optimistically biased by the search itself. Nested CV uses an inner loop for tuning and an outer loop to evaluate the tuned model on data the inner loop never saw, separating selection from evaluation. Trap: reporting the best inner-loop CV score as your final generalization number.
Your model gets 99% training accuracy but only 70% validation accuracy. What is happening and what are your remedies?
A large train-validation gap signals overfitting (high variance): the model memorized training data. Remedies include more or cleaner data, stronger regularization (increase λ), reducing model capacity, early stopping, or dropout. If instead both scores were low, that would be underfitting. Trap: adding more capacity (layers, features) — that worsens overfitting. A suspiciously high validation score suggests leakage, not a breakthrough.
18
🎯E-step: guess who made each point. M-step: refit. Repeat — it only ever climbs.
You have a bag of unlabeled points. Clustering says: find the hidden tribes. EM says: if you knew which tribe each point belongs to, fitting is easy — so guess, fit, guess again, and each round you can only get more right.
K-MEANS: HARD TRIBES
Pick $k$ centroids. Then forever: assign each point to its nearest center, then drag each center to the mean of its points. Two cheap swaps on the same cost — it can only go down.
$$\min_{\{c_k\},\{z_i\}}\sum_{i=1}^{n}\|x_i - c_{z_i}\|_2^2$$total squared distance of every point to its centroid
Assign stepeach point joins nearest centroid (minimizes cost over $z$)
Update stepeach centroid moves to its cluster's mean (minimizes cost over $c$)
k-means++seed new centers far from existing ones — better starts, fewer bad runs
GMM + EM: SOFT TRIBES
Instead of forcing each point into exactly one bucket, let it belong to every cluster with some probability — a responsibility. Then the two steps become: compute responsibilities (E-step), refit each Gaussian weighted by those responsibilities (M-step).
$$p(x)=\sum_{k=1}^{K}\pi_k\,\mathcal{N}(x;\mu_k,\Sigma_k)$$data is a weighted blend of K Gaussian blobs
E-step$\gamma_{ik}=\frac{\pi_k\mathcal{N}(x_i;\mu_k,\Sigma_k)}{\sum_j \pi_j\mathcal{N}(x_i;\mu_j,\Sigma_j)}$ — how much does blob $k$ "own" point $i$?
M-steprefit $\pi_k$, $\mu_k$, $\Sigma_k$ using responsibility-weighted averages
Why it climbseach E-step builds a tight lower bound on log-likelihood; M-step raises it
THE DIAL BETWEEN THEM
k-means is GMM with all covariances fixed to $\sigma^2 I$ and $\sigma \to 0$. As variance shrinks, soft responsibilities collapse to hard 0/1: the point goes to whichever mean is closest. Same algorithm, two ends of one dial.
| k-means | GMM / EM | |
|---|---|---|
| Assignments | Hard (0 or 1) | Soft (probabilities) |
| Cluster shape | Spherical | Elliptical (full $\Sigma_k$) |
| What you get | Labels | Full generative model |
| Converges to | Local optimum of WCSS | Local max of log-likelihood |
| Choose $k$ via | Elbow / silhouette | BIC / AIC |
⚠ Clears up — "EM is a clustering algorithm" EM is a general recipe for maximum-likelihood when latent variables hide the full picture. GMM clustering is just one use. The same recipe fits hidden Markov models, missing-data problems, and much more.
◆ Interview probe "In what sense is k-means a special case of EM?" → Run EM on a GMM where all covariances are $\sigma^2 I$ with equal mixing weights and let $\sigma \to 0$. The soft responsibilities become winner-take-all, and the M-step reduces to moving each centroid to its assigned mean — that is exactly k-means.
Remember EM is the general "guess hidden, refit, repeat" loop that only climbs; k-means and GMM are both special cases, differing only in how hard the assignments are.
Tricky interview questions 10
Walk me through k-means. What is its objective?
Pick $k$ centroids at random; alternate assigning each point to its nearest centroid, then moving each centroid to the mean of its assigned points. This is coordinate descent on within-cluster sum of squared distances (WCSS), so WCSS never increases and the algorithm always converges. Trap: saying it finds the global optimum — it only reaches a local optimum; optimal k-means clustering is NP-hard in general.
Does k-means always converge? To the global optimum?
Yes, it always converges: WCSS can only decrease and there are finitely many assignments, so it must stop. No, not to the global optimum — it reaches a local minimum that depends on initialization. Trap: conflating guaranteed convergence with guaranteed optimality; these are separate facts.
Why is k-means sensitive to initialization, and how does k-means++ fix it?
Different random seeds land in different local optima, some far from good. k-means++ seeds each new centroid with probability proportional to its squared distance from the nearest existing centroid, spreading them out and achieving an expected O(log k) approximation on WCSS. Trap: forgetting to run multiple restarts even with k-means++ — a single run can still be unlucky.
Why must you scale features before running k-means?
k-means uses Euclidean distance, so a feature with a large numeric range dominates the distance computation and therefore dominates cluster structure. Z-score standardization puts all features on equal footing. Trap: treating scaling as always mandatory — if a feature's larger scale genuinely reflects greater importance, blind standardization can wash out real signal.
How do you choose k?
Plot WCSS vs k and look for an elbow (informal); compute the silhouette score (ranges −1 to 1, higher is better separated); use the gap statistic; or for GMMs use BIC or AIC, which penalize model complexity. Trap: minimizing training WCSS directly — it always decreases with more clusters, so you must penalize or hold out data.
What is the EM algorithm and what general problem does it solve?
EM is an iterative MLE method for models with latent (hidden) variables. The E-step computes the expected complete-data log-likelihood under the posterior of the latent variables given current parameters; the M-step maximizes that expectation to update parameters. It is guaranteed to increase (or maintain) the observed-data log-likelihood at every step. Trap: calling it a clustering algorithm — clustering is just one application; EM also fits HMMs, missing-data models, and more.
Why is EM guaranteed to converge, and what does it converge to?
Each E-step constructs a lower bound on the log-likelihood that is tight at the current parameters (via Jensen's inequality); the M-step raises the bound. So the observed log-likelihood increases monotonically at every iteration and must converge. It converges to a local maximum or saddle point, not necessarily the global MLE. Trap: claiming EM finds the global maximum — like k-means it is initialization-dependent.
Walk through the E-step and M-step for a GMM concretely.
E-step: for each point $i$ and component $k$, compute the responsibility $\gamma_{ik}$ = posterior probability the point came from component $k$, using current $\pi_k, \mu_k, \Sigma_k$. M-step: update $\pi_k$ to the average responsibility, $\mu_k$ to the responsibility-weighted mean of the data, and $\Sigma_k$ to the responsibility-weighted scatter matrix. Trap: reversing the steps or saying M-step assigns points — only E-step computes memberships.
In what sense is k-means a special case of GMM/EM?
k-means is the limit of EM on a GMM with equal mixing weights and covariances fixed to $\sigma^2 I$ as $\sigma \to 0$. The soft responsibilities collapse to hard 0/1, giving the nearest-centroid assignment, and the M-step becomes the plain mean update. Trap: stating they are "related" without the mechanism — the key is the shrinking spherical covariance that turns soft into hard.
What are the failure modes of EM for GMM, and how do you prevent them?
A component can collapse onto a single point, driving its variance to zero and the likelihood to infinity (singularity). EM also stalls at poor local optima. Fix with covariance regularization (a small diagonal floor), restricting covariance type (spherical/diagonal/tied), k-means++ initialization, and multiple restarts. Trap: not knowing the singularity exists — with unrestricted covariances, maximum likelihood is ill-posed and regularization is not optional.
19
🎯PCA spins your axes to face the variance: keep the long directions, drop the flat ones.
PCA spins your coordinate axes until the first axis points at the widest cloud of data, the second at the next widest, and so on — then you throw away the skinny axes. ICA goes further: it unscrambles axes until the signals along them are statistically independent, like separating two voices recorded by the same microphone.
PCA: VARIANCE FIRST
Center your data (subtract the mean). The principal components are the eigenvectors of the covariance matrix, ranked by eigenvalue. Eigenvalue = variance captured along that direction. Project onto the top k eigenvectors and you have a k-dimensional summary that throws away as little spread as possible.
$$\Sigma\, u_j = \lambda_j\, u_j, \quad \lambda_1 \ge \lambda_2 \ge \cdots$$covariance matrix × principal direction = eigenvalue × same direction
Fraction of variance kept with k components: sum the top k eigenvalues, divide by the total (trace of the covariance).
HOW YOU ACTUALLY COMPUTE IT: SVD
Never form XᵀX explicitly — it squares the condition number and kills numerical precision. Instead, take the SVD of the centered data matrix directly.
$$X = U S V^\top$$columns of V are principal directions; diagonal of S² / (n−1) are the eigenvalues
The right singular vectors (columns of V) are the principal components. Every PCA library does this under the hood.
ICA: INDEPENDENCE, NOT VARIANCE
PCA only removes linear correlation. ICA seeks directions where the projected signals are statistically independent — not just uncorrelated. Classic use: cocktail-party problem. Three microphones, three overlapping speakers — ICA recovers the three clean voices. Unlike PCA, ICA has no natural ranking of components and assumes non-Gaussian sources (Gaussians have no higher-order structure to exploit).
PCA goalmaximize variance (compress, denoise, visualize)
ICA goalmaximize statistical independence (source separation)
PCA componentsorthogonal, uncorrelated, ranked by eigenvalue
ICA componentsindependent (non-Gaussian), no natural rank order
Labels needed?neither — both are fully unsupervised
| PCA | t-SNE / UMAP | |
|---|---|---|
| Structure | linear | nonlinear |
| Use for | features, denoising, any downstream task | 2-D / 3-D visualization only |
| Preserves | global variance | local neighborhoods |
| Invertible? | yes | no |
⚠ Clears up — PCA is not feature selection Each principal component is a weighted mix of all original features, not a chosen subset. Feature selection picks columns; PCA rotates and blends them. Calling PCA "feature selection" is one of the most common interview slip-ups.
◆ Interview probe "Your manager says to remove highly correlated features before PCA to clean the data — good idea?" → No. Correlation is exactly what gives PCA room to compress. Correlated features collapse into a single component; removing them first discards information PCA would have used. Just standardize, then let PCA do its job.
Remember PCA rotates to variance — keep the wide axes, drop the flat ones; ICA rotates to independence — separate the mixed signals.
Tricky interview questions 10
What is PCA and how does it differ from feature selection?
PCA is a linear dimensionality reduction technique that finds the directions of maximum variance in the data and projects onto a lower-dimensional subspace. Each component is a weighted linear combination of all original features — not a chosen subset. Trap: calling PCA "feature selection." Feature selection keeps original columns; PCA creates new axes that blend all of them.
What do eigenvalues and eigenvectors of the covariance matrix represent in PCA?
Eigenvectors are the principal directions — the axes you rotate to. Each paired eigenvalue is the variance of the data projected onto that direction. Components are sorted by eigenvalue, largest first, so the first component captures the most spread. Trap: swapping them — the eigenVECTOR is the direction; the eigenVALUE is the scalar variance magnitude.
Why must you center (and usually scale) data before PCA?
Centering (subtracting the mean) is mandatory: without it the first component partly captures the data's offset from the origin rather than its variance structure. Scaling to unit variance is needed whenever features have different units, because PCA maximizes variance and a large-scale feature will dominate otherwise. Trap: confusing the two — centering is always required; scaling depends on whether units are comparable.
How do you choose k, the number of components to keep?
Plot the cumulative explained-variance ratio and choose k where it crosses a threshold (often 90–95%), or find the scree-plot elbow. For supervised tasks, tune k on a validation set because high variance does not mean high predictive power. Trap: treating 95% as universal — for visualization k is 2 or 3; for a downstream model, validation should drive the choice.
How is PCA related to SVD, and why is SVD preferred computationally?
The right singular vectors of the centered data matrix X are exactly the principal components, and the squared singular values equal the eigenvalues times (n−1). SVD avoids forming XᵀX explicitly, which would square the condition number and amplify numerical errors. Trap: thinking SVD and eigendecomposition of the covariance are separate algorithms — they solve the same problem; SVD is the stable implementation.
Are principal components uncorrelated? Are they independent?
Yes, they are linearly uncorrelated — the covariance matrix is symmetric, so its eigenvectors are orthogonal, making projections uncorrelated. They are not necessarily statistically independent unless the data is Gaussian. Independence (higher-order structure) is what ICA targets. Trap: equating uncorrelated with independent — uncorrelated only rules out linear dependence.
You ran PCA before a classifier and accuracy dropped. What likely went wrong?
PCA is unsupervised and maximizes variance, not class separability. Likely causes: the discriminative signal lives in low-variance directions that PCA discarded, too few components were kept, or PCA was fit on the full dataset before splitting (data leakage). Fix: fit PCA inside cross-validation on training folds only, and tune k. Trap: assuming PCA always helps or fitting it on the full dataset before the train/test split.
When does PCA fail or give misleading results?
PCA fails when structure is nonlinear (a curved manifold), when the useful signal lies in low-variance directions, when all eigenvalues are roughly equal (isotropic variance, no preferred direction), or when outliers distort the covariance. It also gives no benefit when features are already uncorrelated. Trap: assuming high variance always equals high information — for classification, the discriminative direction can be low-variance.
How does PCA compare to LDA for dimensionality reduction?
PCA is unsupervised and maximizes total variance; LDA is supervised and maximizes the ratio of between-class to within-class scatter. LDA can yield at most (C−1) components where C is the number of classes. They can produce completely different projections on the same data. Trap: saying both maximize variance — LDA maximizes class separability using labels, which PCA ignores entirely.
How can PCA be used for anomaly detection?
Project each point onto the top-k components and reconstruct it back to the original space. Normal points lie near the principal subspace and have small reconstruction error; anomalies don't fit the dominant variance structure and have large error. Flag points whose reconstruction error exceeds a threshold. Trap: forgetting that outliers distort the covariance matrix used to fit PCA — robust PCA or prior outlier removal is often needed.
20
🎯A tree is just 20-questions on your features — it carves space into axis-aligned boxes.
A decision tree plays twenty questions with your data: at each step it asks one yes/no question about a single feature ("is age < 30?"), and keeps slicing the space into rectangular boxes until each box is mostly one answer.
HOW IT GROWS — GREEDY SPLITTING
Top-down and greedy. At every node it scans all feature/threshold pairs and keeps the one split that makes the two children as pure as possible, then recurses. It never backtracks, so an early bad question is stuck forever.
"Best" = largest drop in impurity, the information gain:
$$\text{gain}=I(\text{parent})-\sum_{\text{child}}\frac{n_{\text{child}}}{n_{\text{parent}}}\,I(\text{child})$$parent impurity minus the size-weighted impurity of the children — pick the split that maximizes this
IMPURITY — HOW MIXED IS A NODE
Both measures are max when classes are 50/50 and 0 when the node is pure. $p_k$ is the fraction of class $k$ in the node.
$$\text{Gini}=1-\sum_k p_k^{2}\qquad H=-\sum_k p_k\log p_k$$Gini = chance two random picks disagree; Entropy = bits of surprise — both just score "how mixed"
| Gini | Entropy | |
|---|---|---|
| cost | cheaper (no log) | has a log |
| resulting tree | nearly identical in practice — choice rarely matters | |
REGRESSION TREES
Same machine, swap the impurity. Now a "pure" leaf is one with low spread, so a split minimizes variance / SSE in the children. The leaf predicts the mean of its training targets.
$$I_{\text{node}}=\sum_{i\in\text{node}}(y_i-\bar y)^2$$sum of squared distances to the leaf mean — the regression "impurity"
predictionclassification → majority class of the leaf; regression → mean of the leafthe modela set of axis-aligned boxes tiling feature space, each box one constant output
WHY WE LOVE THEM / WHY WE DON'T
| Pros | Cons |
|---|---|
| readable rules (a flowchart) | HIGH VARIANCE — nudge the data, get a totally different tree |
| non-linear, captures interactions | greedy → only locally optimal |
| mixed feature types, no scaling | overfits if grown full |
| handles missing-ish data well | splits are axis-aligned only (staircase, never diagonal) |
Tame overfitting with max depth, min samples per leaf, and cost-complexity pruning (grow full, then snip weak branches penalized by a leaf count $\alpha$). Feature importance = total impurity reduction a feature contributes across all its splits.
⚠ Clears up — no feature scaling A split only asks "is $x$ above or below a threshold?", which is invariant to any monotonic rescaling. So standardizing or min-max scaling does nothing to a tree — unlike KNN, SVM, or linear models where it's essential.
⚠ Clears up — interactions, but only staircases Each root-to-leaf path is an AND of several conditions, so trees do model feature interactions and non-linearity for free. But every boundary is axis-aligned, so a true diagonal boundary is approximated by a jagged staircase.
◆ Interview probe "Does a tree find the optimal split structure?" → No. Each split is locally optimal (greedy); finding the globally optimal tree is NP-hard, so a poor early split can't be undone.
Remember One tree = greedy purity-cutting boxes — interpretable but high variance, which is exactly the itch we scratch by ensembling next.
Tricky interview questions 11
How does a decision tree decide where to split?
It greedily scans all feature/threshold pairs and keeps the split with the largest impurity drop — information gain (entropy) or Gini for classification, variance/MSE reduction for regression — then recurses top-down. Trap: saying it finds the globally optimal tree; greedy is only locally optimal and the optimal tree is NP-hard.
What is information gain, exactly?
$\text{gain}=I(\text{parent})-\sum\frac{n_{\text{child}}}{n_{\text{parent}}}I(\text{child})$ — the parent's impurity minus the size-weighted impurity of its children. The split with the biggest gain wins. Trap: forgetting the size weighting and averaging children equally.
Gini vs entropy — does it matter which you use?
Both are 0 when pure and maximal when classes are balanced; Gini $=1-\sum p_k^2$, entropy $=-\sum p_k\log p_k$. They produce nearly identical trees; Gini is slightly cheaper (no log). Trap: claiming one is clearly superior or gives very different trees — in practice it rarely matters.
How does a regression tree split and predict?
It chooses the split that minimizes variance / SSE in the children, and each leaf predicts the mean of its training targets. The output surface is piecewise-constant boxes. Trap: reusing Gini/entropy for regression — those are classification impurities; regression uses squared error.
Why do trees overfit, and how do you prevent it?
A fully grown tree splits until leaves are pure, memorizing noise — low bias, very high variance. Control it with pre-pruning (max_depth, min_samples_leaf, min_samples_split), post-pruning (cost-complexity), or ensembling. Trap: confusing pre-pruning (early stopping) with post-pruning (grow full, then prune).
Do you need to scale or normalize features for trees?
No. Splits compare a feature to a threshold, which is invariant to any monotonic rescaling, so standardization has no effect. Trap: saying "all ML needs scaling" — that's true for KNN, SVM, linear/logistic and nets, not trees.
Can a single tree capture non-linearity and feature interactions?
Yes — each root-to-leaf path is a conjunction of conditions, so depth $d$ allows up to $d$-way interactions and non-linear boundaries, no manual engineering. Trap: expecting smooth or diagonal boundaries — trees are piecewise-constant and axis-aligned (a staircase approximating any diagonal).
What's the geometry of a tree's decision boundary?
A set of axis-aligned rectangular boxes tiling feature space; each box outputs one constant (majority class or leaf mean). Trap: drawing a diagonal — a single split can only cut perpendicular to one axis.
How is feature importance computed in a tree, and what's the pitfall?
Default (MDI/Gini) importance sums the impurity reduction a feature contributes across all its splits. It's biased toward high-cardinality and continuous features (more split points) and is computed on training data. Trap: trusting it blindly — prefer permutation importance or SHAP on held-out data.
Why is a decision tree considered a high-variance model?
Because the greedy split at the top depends sharply on the exact training rows — a small data change can flip an early split and cascade into a completely different tree (and predictions). Trap: calling a deep tree high-bias; an unconstrained tree is low-bias, high-variance.
Given trees are so unstable, why use them at all?
They're interpretable, non-linear, handle mixed feature types with no scaling, and are fast. Their high variance is the very weakness that averaging many trees (bagging/forests) and boosting are designed to fix. Trap: treating high variance as fatal — it's the launchpad for ensembles.
21
🎯Bag to calm the jitter (variance); boost to fix the aim (bias).
Many mediocre opinions, combined right, beat one expert. Bagging polls independent voters to cancel out their noise (kills variance); boosting drafts a relay team where each runner fixes the last one's mistakes (kills bias).
BAGGING — AVERAGE AWAY THE NOISE
Train the same high-variance learner on many bootstrap resamples (sample with replacement), then average (regression) or vote (classification). Each tree overfits differently, so their errors cancel. Bias is unchanged — you only smooth out the wobble.
$$\operatorname{Var}\!\left(\tfrac{1}{N}\sum_i T_i\right) = \rho\,\sigma^2 + \frac{1-\rho}{N}\,\sigma^2$$averaging $N$ trees, each variance $\sigma^2$, pairwise correlation $\rho$: only the $(1-\rho)/N$ part shrinks — correlation $\rho$ sets the floor
If trees were independent ($\rho=0$) variance drops to $\sigma^2/N$. They never are, so the real lever is lowering $\rho$. Needs high-variance base learners (deep, unpruned trees) — averaging already-stable models does nothing.
RANDOM FOREST — DECORRELATE THE TREES
Bagging's weakness: one dominant feature makes every tree split the same way, so $\rho$ stays high. Fix: at each split, only let the tree choose from a random subset of features (~$\sqrt{p}$ for classification). This forces variety, drives $\rho$ down, and squeezes out more variance reduction.
OOB erroreach row is left out of ~$1/e \approx 37\%$ of trees; predict it with only those — free held-out estimate, no CV neededFeature importanceimpurity drop (MDI) or permutation importance per feature
⚠ Clears up — RF is more than bagging Plain bagging samples rows; random forest also samples columns at every split. That per-split feature randomness is the whole point — without it the trees stay correlated and averaging barely helps.
BOOSTING — FIX MISTAKES IN SEQUENCE
Train weak learners (shallow trees / stumps) one after another; each new one focuses on what the ensemble got wrong so far. You build a strong learner from weak ones, attacking bias. The catch: it's sequential and can overfit if you run too long.
AdaBoost: after each round, up-weight the misclassified points so the next stump cares about them. This is exactly forward-stagewise fitting of an exponential loss.
Gradient boosting (GBM / XGBoost / LightGBM): each new tree fits the negative gradient of the loss w.r.t. current predictions — gradient descent in function space.
$$F_{m}(x) = F_{m-1}(x) + \nu\, h_m(x), \qquad h_m \approx -\frac{\partial L}{\partial F_{m-1}}$$add tree $h_m$ fit to the negative gradient, scaled by learning rate $\nu$ (shrinkage)
For squared-error loss the negative gradient is the residual — that's why "fit the residuals" is just the MSE special case. Tame overfitting with shrinkage ($\nu$ small + more trees), row/column subsampling, L1/L2 on leaf weights, and early stopping. XGBoost adds second-order (Hessian) split scoring and learned default directions for missing values; histogram splits (XGBoost/LightGBM) bucket features for speed — the default winner on tabular data.
| Bagging / Random Forest | Boosting | |
|---|---|---|
| Training | parallel, independent | sequential, dependent |
| Attacks | variance | bias |
| Base learner | strong, deep trees | weak, shallow trees / stumps |
| More trees | never hurts (plateaus) | can overfit → early-stop |
| How it combines | average / majority vote | weighted additive sum |
| Robust to noise | yes | more sensitive |
STACKING — A META-LEARNER ON TOP
Train several different base models, then feed their out-of-fold predictions into a small meta-learner that learns how to best blend them. Bagging/boosting reuse one learner type; stacking combines diverse ones.
◆ Interview probe "Can adding more trees hurt a random forest?" → No — extra trees only stabilize the average, so RF never overfits in the number of trees. In boosting it can, because each tree fits residuals, so too many keeps chasing noise — hence early stopping.
Remember Bagging = parallel, decorrelate, kill variance; Boosting = sequential, fix errors, kill bias.
Tricky interview questions 12
What is the difference between bagging and boosting?
Bagging trains many high-variance models independently in parallel on bootstrap samples and averages them, reducing variance. Boosting trains weak learners sequentially, each correcting the previous ones' errors, reducing bias. Trap: reversing them, or forgetting that bagging uses strong deep trees while boosting uses weak shallow ones.
Why does bagging reduce variance but not bias?
Averaging $N$ predictors with correlation $\rho$ gives variance $\rho\sigma^2 + \tfrac{1-\rho}{N}\sigma^2$, so the noise shrinks while each tree's bias is left untouched. Trap: claiming bagging reduces bias — it doesn't; and correlation $\rho$ caps how far variance can drop.
How does a random forest differ from a single tree, and why is it better?
RF is bagging of trees plus a random feature subset at each split. Bootstrap rows and per-split feature randomness decorrelate the trees, so averaging cuts variance sharply with little bias cost. Trap: calling RF "just bagging" — the per-split feature subsampling ($\sim\sqrt{p}$) is what decorrelates the trees.
What is out-of-bag (OOB) error?
Each bootstrap sample leaves out about $1/e \approx 37\%$ of rows; each example is scored only by trees that didn't see it, and aggregating gives a near-free cross-validation estimate. Trap: forgetting the ~1/3 left-out fraction, or applying OOB to boosting — it's a bagging-only concept.
What does "gradient" mean in gradient boosting?
Each new tree is fit to the negative gradient of the loss w.r.t. the current model's predictions — gradient descent in function space. For squared-error loss those gradients equal the residuals. Trap: thinking it always fits residuals; residuals are the gradient only for MSE — other losses fit generalized pseudo-residuals.
Compare random forest and gradient boosted trees, and when to use each.
RF builds deep trees in parallel and averages (variance-down, robust, easy to tune); GBDT builds shallow trees sequentially fitting gradients (bias-down, higher accuracy but tuning-sensitive). Use RF for a robust baseline, GBDT for max accuracy when you can tune. Trap: saying GBDT is always better — it's noise-sensitive, and more trees can hurt it but never hurt RF.
Why shallow trees in boosting but deep trees in random forests?
Boosting attacks bias by sequentially correcting errors, so it starts from high-bias/low-variance weak learners. Bagging/RF attacks variance, so it starts from low-bias/high-variance deep trees and averages them down. Trap: mixing it up — deep trees in boosting overfit fast; shallow trees in a forest underfit.
What makes XGBoost different from classic GBM?
It adds L1/L2 regularization on leaf weights plus a leaf-count penalty, uses a second-order Taylor (gradient + Hessian, Newton boosting) to score splits, supports row/column subsampling, learns default directions for missing values, and adds engineering speedups (sparsity-/cache-aware, parallel split finding, histograms). Trap: saying it's "just faster" — the bigger wins are the regularized objective and second-order info.
How does XGBoost handle missing values?
It doesn't impute. At each split it learns a default direction by trying both branches and picking the higher-gain one; missing rows are routed there at inference. Trap: saying it imputes mean/median or that you must impute first — it learns the optimal default branch from data.
How do learning rate and number of trees interact in boosting?
The learning rate scales each tree's contribution; smaller learns slower but generalizes better, needing more trees. Best practice: low learning rate plus more trees with early stopping. Trap: thinking more trees always helps — in boosting too many overfit; in random forests more trees never overfit.
How is feature importance computed in tree ensembles, and what's the catch?
Default MDI/Gini importance sums each feature's impurity reduction across splits, but it's biased toward high-cardinality and continuous features and is computed on training data. Permutation importance or SHAP on held-out data is more reliable. Trap: trusting default impurity importances blindly — they inflate high-cardinality features and reflect training fit, not generalization.
What is stacking, and how does it differ from bagging/boosting?
Stacking trains several diverse base models, then a meta-learner blends their out-of-fold predictions into a final output. Bagging and boosting reuse one learner type to fight variance or bias respectively; stacking combines heterogeneous models. Trap: training the meta-learner on in-sample base predictions — you must use out-of-fold predictions or it leaks and overfits.
22
🎯Bellman in one breath: value now = reward now + discounted value next.
An agent tries actions, the world pushes back with rewards, and the agent gradually learns which choices lead to the most reward over time — like a puppy figuring out which tricks earn treats, purely by trial and error.
THE FORMAL WORLD: MDP
Every RL problem is an MDP — a tuple (S, A, P, R, γ). "Markov" is the key word: the next state depends only on the current state and action, not on the full history. The current state is all you need.
Sstates — all situations the agent can be in
Aactions — the choices available
P(s′|s,a)transition — how the world responds
R(s,a)reward — the only training signal
γ ∈ [0,1)discount — how much future reward is worth now
A policy π(a|s) is just the agent's decision rule: given this state, pick this action.
THE GOAL AND VALUE FUNCTIONS
The agent wants to maximize discounted return — sum all future rewards but shrink distant ones by γt. To plan, we grade every state or state–action pair with a value function.
$$V^*(s) = \max_a \bigl[R(s,a) + \gamma\,\mathbb{E}[V^*(s')]\bigr]$$Best value now = best reward now + discounted best value next
V*(s) grades a state. Q*(s,a) grades a state–action pair. Once you have Q*, the optimal policy is trivial: always pick the action with the highest Q.
BELLMAN: THE RECURSIVE TRICK
The Bellman equation turns an infinite-horizon sum into a one-step recursion. Instead of summing rewards all the way to the end, you say: value now = reward now + discounted value next. Iterate this until convergence — that's value iteration. Q-learning does the same thing online, updating one experience at a time.
| Value Iteration | Q-Learning | |
|---|---|---|
| Needs model? | Yes (P is known) | No (model-free) |
| Updates | Sweep all states | One experience at a time |
| Convergence | Exact, in tabular | Approximate (with function approx.) |
EXPLORATION VS EXPLOITATION
The fundamental tension: do you exploit what you know (pick the best action so far) or explore to find something better? ε-greedy is the classic fix — be greedy most of the time, but pick a random action with probability ε. Without exploration you get stuck; without exploitation you never profit.
⚠ Clears up — discount γ is not optional γ < 1 is required for the infinite sum to converge to a finite number. Setting γ = 1 means all future rewards count equally — fine if episodes always end, but the math breaks for infinite-horizon tasks.
◆ Interview probe Why does Q-learning converge to the optimal policy even if the policy used to collect data is not optimal? → Because Q-learning is off-policy: the target uses max over actions regardless of what the behavior policy did. It learns the greedy policy's value while exploring freely.
Remember Bellman says: value now = reward now + discounted value next — iterate that one-step trick to optimality.
Tricky interview questions 10
What does the Markov property mean in an MDP, and when does it fail?
The next state depends only on the current state and action, not on history. It fails when the observation is partial — e.g., in poker you cannot see opponents' cards, so the observed state does not fully summarize history. The fix is a Partially Observable MDP (POMDP), which maintains a belief distribution over true states. Trap: assuming the Markov property always holds once you define "state" — in practice states are observations, and observations are often incomplete.
What is the difference between V*(s) and Q*(s,a), and which is more useful for choosing actions?
V*(s) is the expected return starting from state s under the optimal policy. Q*(s,a) is the same but you commit to action a first, then act optimally. Q* is more useful for action selection: π*(s) = argmax_a Q*(s,a) requires no model of transitions, whereas extracting a greedy action from V* alone requires knowing P(s′|s,a) to evaluate one-step lookahead. Trap: thinking V* is sufficient without a model — you also need the transition dynamics to act on it.
Explain the difference between on-policy and off-policy learning. Which is Q-learning?
On-policy methods (e.g., SARSA) learn the value of the policy they are currently following, so exploration affects what is learned. Off-policy methods (e.g., Q-learning) learn the optimal policy's value regardless of the behavior policy used to collect data. Q-learning is off-policy: the TD target uses max_a Q(s′,a) regardless of which action was actually taken. Trap: confusing off-policy with "no exploration needed" — you still need exploration to visit all states, but learning targets the greedy policy.
What is temporal-difference (TD) learning and how does it differ from Monte Carlo?
TD learning updates value estimates after every step using bootstrapping: V(s) ← V(s) + α[r + γV(s′) − V(s)]. Monte Carlo waits until the episode ends and uses the actual full return. TD has lower variance (no need to wait) but introduces bias from the bootstrap estimate; MC is unbiased but high-variance and cannot handle continuing (non-episodic) tasks. Trap: calling TD "better" — it trades bias for variance; the right choice depends on episode length and whether bootstrapping introduces harmful bias.
What problem does the discount factor γ solve, and what happens if γ → 1?
γ < 1 makes the infinite sum of future rewards converge to a finite value and encodes a preference for near-term reward. As γ → 1 the agent becomes far-sighted; at γ = 1 the sum diverges for continuing tasks, breaking the math. For episodic tasks γ = 1 is acceptable since the sum is finite by episode termination. Trap: using γ = 1 in a continuing-task setup — this makes value functions undefined unless the environment is guaranteed to terminate.
What is the vanishing gradient problem in deep RL (or deep nets generally), and how is it addressed?
Backprop multiplies many small derivatives, shrinking gradients to near zero in early layers so learning stalls. Fixes: ReLU activations (gradient = 1 for positive inputs), proper initialization (He/Xavier), batch normalization, and residual/skip connections. In RL specifically, reward sparsity compounds the problem. Trap: confusing vanishing gradients with exploding gradients, or thinking a smaller learning rate helps — the issue is the gradient signal decaying through depth, not the step size.
Why is exploration critical in RL but not in supervised learning?
In supervised learning the dataset is fixed; the model does not choose which examples it sees. In RL the agent generates its own data by acting, so if it never explores, it never visits states needed to learn their value and can get stuck in a locally good but globally suboptimal policy. Trap: proposing a purely greedy policy from the start — without exploration it cannot discover that a different sequence of actions yields higher long-run reward.
What is the "deadly triad" in RL and why does it cause instability?
The deadly triad is the combination of function approximation (e.g., a neural net), bootstrapping (TD-style targets), and off-policy learning. Together they can cause divergence because the bootstrap target itself changes as weights update, and off-policy sampling creates a distribution mismatch. DQN stabilizes this with a target network and replay buffer. Trap: thinking any one of the three alone is dangerous — each is fine individually; instability emerges from their combination.
How does random forest differ from Q-learning in terms of bias-variance tradeoff?
Random forest reduces variance by averaging many decorrelated trees trained in parallel on bootstrap samples — it is a variance-reduction technique. Q-learning reduces bias through iterative Bellman updates that asymptotically converge to the true Q* — it is a bias-reduction process over time. Both can overfit but via different mechanisms: RF through noisy averaging, Q-learning through function approximation. Trap: conflating the two frameworks — they solve entirely different problems (supervised prediction vs. sequential decision-making).
When would you prefer model-based RL over model-free RL?
Model-based RL (learning or using the transition dynamics P(s′|s,a)) is preferred when data is expensive or scarce — a learned model lets you plan with simulated rollouts, dramatically improving sample efficiency. Model-free is preferred when the environment is complex enough that learning an accurate model is harder than directly learning a policy or value function (e.g., high-dimensional pixel inputs). Trap: always defaulting to model-free because it is simpler — in robotics or drug discovery where real data is costly, model-based methods can be orders of magnitude more efficient.
23
🎯AUC says how well you rank; calibration says whether to believe the number.
A model has two jobs: rank the right things higher, and tell the truth about its own confidence. AUC measures the first; calibration measures the second — and you need both.
RANKING VS BELIEVING
AUC is threshold-free: it equals the probability that the model scores a random positive above a random negative. That is pure rank quality — it says nothing about whether a "0.7" actually means 70%.
On rare-event problems (fraud, cancer, 1% positives) ROC looks rosy because FPR has the enormous negative pool in its denominator. A handful of false positives barely moves it. The PR curve is honest: precision crashes toward the base rate $\pi$ the moment you generate false positives, so it shows what ROC hides.
$$\text{AUC} = P(\hat p^+ > \hat p^-)$$probability the model ranks a positive above a negative
CALIBRATION — DOES THE NUMBER MEAN ANYTHING?
A reliability diagram plots predicted probability (x) vs observed frequency (y). A perfect model lies on the diagonal. The gap off the diagonal, averaged over bins weighted by how many predictions fall there, is the ECE.
$$\text{ECE} = \sum_m \frac{|B_m|}{n}\,|\text{acc}(B_m) - \text{conf}(B_m)|$$average confidence gap, weighted by bin size
Fix miscalibration with Platt scaling (logistic layer on top of scores), isotonic regression (non-parametric), or temperature scaling (one parameter: divide logits by T). None of these change AUC — they only bend the probability curve back onto the diagonal.
PRECISION / RECALL / F1 IN ONE BREATH
PrecisionOf what I called positive, how many are right? — TP/(TP+FP). Raise threshold → precision up.
RecallOf all true positives, how many did I catch? — TP/(TP+FN). Raise threshold → recall down.
F1Harmonic mean 2PR/(P+R). Dominated by the smaller one, so both must be high.
F-betabeta > 1 favors recall; beta < 1 favors precision. Use when costs are asymmetric.
| Optimize precision when | Optimize recall when |
|---|---|
| FP is costly (auto-block, ads) | FN is costly (cancer, fraud review) |
REGRESSION METRICS FAST
MAEMean absolute error — linear penalty, robust to outliers, targets conditional median.
RMSERoot MSE — squares errors, punishes big mistakes, targets conditional mean. RMSE ≥ MAE always.
R²Fraction of variance explained; never decreases when you add features — use adjusted R² to penalize complexity.
⚠ Clears up — high AUC does not mean calibrated A model can rank perfectly (AUC 0.99) yet output probabilities that are wildly off. AUC measures order; calibration measures magnitude. Always check both, especially when the probability itself drives a decision (bid price, risk threshold).
◆ Interview probe "Your fraud model has AUC 0.99 but operations says the alerts are useless." → Check precision at the deployed threshold — high AUC under 1% prevalence still allows terrible precision. Pull the PR curve, pick a threshold that balances alert volume against review capacity, and verify calibration so expected-loss calculations hold.
Remember AUC tells you who wins the race; calibration tells you whether to believe the finish-line clock.
Tricky interview questions 10
On a highly imbalanced dataset (1% positives), should you use ROC-AUC or PR-AUC, and why?
Prefer PR-AUC. ROC's FPR has the huge negative count in its denominator, so even many false positives barely move FPR, making ROC-AUC look optimistically high. Precision puts FP against predicted positives, directly exposing the false-positive problem that matters in rare-event detection. The PR baseline equals prevalence, not 0.5, so a "good" PR-AUC is dataset-dependent. Trap: defaulting to ROC-AUC because it is "standard" without checking class imbalance.
What is the probabilistic interpretation of AUC?
AUC equals P(score of a random positive > score of a random negative). It is a pure ranking metric, threshold-free and prevalence-independent. Trap: thinking AUC measures accuracy or calibration — a model can have AUC 0.95 with badly miscalibrated probabilities.
What is model calibration, and how does it differ from discrimination?
Calibration means predicted probabilities match observed frequencies: among samples scored 0.7, about 70% are truly positive. Discrimination (AUC) only measures rank order. A perfectly ranking model can still be miscalibrated, and recalibrating via Platt scaling or temperature scaling does not change AUC. Trap: assuming a high-AUC model gives trustworthy probabilities for decisions like pricing or risk thresholds.
Why is F1 the harmonic mean rather than the arithmetic mean of precision and recall?
The harmonic mean is dominated by the smaller value, so F1 is only high when both precision and recall are high. An arithmetic mean would let a model hide terrible recall behind great precision. Trap: treating F1 as a safe universal default — it weights P and R equally and ignores true negatives, so use F-beta when costs are asymmetric.
Why is accuracy a poor metric for imbalanced classification?
With 99% negatives, a model that always predicts "negative" scores 99% accuracy while catching zero positives. Accuracy treats all errors equally and is dominated by the majority class. Use precision, recall, F1, or PR-AUC instead. Trap: quoting high accuracy as evidence of a good model without checking class balance or the confusion matrix.
How does changing the decision threshold affect precision and recall?
Raising the threshold makes the model more conservative: precision usually rises (fewer FP) while recall falls (more missed positives), and vice versa. The threshold is the operating point you tune using the PR or ROC curve based on the relative cost of FP vs FN. Trap: assuming the default 0.5 threshold is optimal — it rarely is for imbalanced or cost-asymmetric problems.
Compare RMSE and MAE. When do you prefer each?
RMSE squares errors before averaging, penalizing large errors disproportionately and targeting the conditional mean; MAE weights all errors equally, is more robust to outliers, and targets the conditional median. Use RMSE when large errors are especially bad; use MAE when outliers should not dominate. Trap: claiming they differ by a constant — RMSE is always ≥ MAE and the gap grows with error variance.
What does R-squared measure, and what problem does adjusted R-squared fix?
R-squared is 1 minus the ratio of residual variance to total variance — the fraction of target variance explained. It never decreases when you add features, even useless ones. Adjusted R-squared penalizes the number of predictors and increases only if a new feature helps more than chance would. Trap: treating high R-squared as proof of a good model — it can be inflated by overfitting and can go negative on a test set.
What is the difference between macro, micro, and weighted averaging for multi-class F1?
Macro computes F1 per class and takes an unweighted mean, so rare classes count equally. Micro pools all TP/FP/FN across classes, so it is dominated by frequent classes (and equals accuracy for single-label problems). Weighted scales each class score by its support. Trap: using micro-F1 on imbalanced data when you care about rare classes — it masks poor minority-class performance.
Why might an offline metric improve while the online A/B-test metric does not?
Offline metrics are proxies on logged data with selection and feedback bias; they cannot capture novelty, presentation effects, or how new recommendations change user behavior. The online A/B test measures real causal effects on live users, so the two can diverge. Trap: treating an offline win as ship-ready — use offline metrics to filter candidates, then require an A/B test on the north-star metric to confirm a real gain.
24
🎯If the answer isn’t instant, that’s the chapter to reread.
A recognition drill, not new material: a stack of one-line interview questions, each with a two-sentence answer and the chapter it lives in. Read a question, answer it in your head before you read the answer — if it comes out instantly you own that chapter; if you stall, that hesitation is the page pointing you back to the source.
Modeling choices — what kind of model, and why
Generative vs discriminative? (ch9)
Discriminative models the boundary directly, $p(y\mid x)$; generative models how the data is made, $p(x\mid y)\,p(y)$, then flips it with Bayes. Generative is more data-efficient when its assumptions hold (e.g. GDA with Gaussians beats logistic regression in the small-data regime), but discriminative wins when they don't.
Why softmax + cross-entropy? (ch7, ch8)
It's the MLE of a categorical distribution — softmax is the exponential-family canonical link, so $-\log p_{\text{true}}$ is the natural loss. The objective is convex in the logits with a clean gradient $\hat p-y$, so optimization is well-behaved.
Kernels vs explicit features? (ch10)
The kernel trick computes inner products $\langle\phi(x),\phi(z)\rangle=k(x,z)$ in a high- or infinite-dimensional feature space without ever forming $\phi$. The RBF kernel $k(x,z)=\exp(-\lVert x-z\rVert^2/2\sigma^2)$ corresponds to an infinite-dimensional $\phi$ — explicit features could never match it.
Regularization & conditioning
Why is L1 sparse and L2 not? (ch4, ch13)
The L1 ball has corners on the axes, so the constrained optimum tends to land exactly on a corner where some weights are 0; equivalently L1's subgradient is a constant $\pm\lambda$ that keeps pushing weights to 0. L2's ball is round and its gradient $2\lambda\theta$ vanishes as $\theta\to0$, so it only shrinks, never zeroes.
Collinearity — what breaks? (ch6)
Highly correlated features make $X^\top X$ nearly singular, so OLS $\hat\theta=(X^\top X)^{-1}X^\top y$ has huge, unstable, high-variance coefficients. Ridge fixes it: $(X^\top X+\lambda I)$ is always invertible and well-conditioned.
MLE properties? (ch3, ch4)
Under regularity, the MLE is consistent (→ true $\theta$) and asymptotically efficient & normal (hits the Cramér–Rao bound, $\hat\theta\approx\mathcal{N}(\theta,\,I(\theta)^{-1}/n)$). But with finite data it can overfit — so add a prior and use MAP.
Diagnosis & evaluation
Why not just minimize training error? (ch5, ch13)
Training error is an optimistically biased estimate of test error — you can drive it to 0 by memorizing, which is overfitting. Pick complexity by cross-validation (or a held-out set), which estimates the quantity you actually care about.
Symptom → bias or variance? (ch5)
Add more data (or look at the gap between train and val error): if the gap closes, it was variance; if both errors stay high and more data doesn't help, it's bias (underfitting) and you need a richer model or better features.
Precision vs recall; ROC vs PR? (ch18)
Precision = of those flagged positive, how many are right; recall = of the true positives, how many you caught. Under heavy class imbalance prefer PR curves (ROC's TPR/FPR look deceptively good when negatives dominate), and pick the operating threshold by the relative cost of false positives vs false negatives.
Ensembles, optimization & worldview
Bagging vs boosting? (ch16)
Bagging trains independent models on bootstrap samples and averages — this cuts variance and is embarrassingly parallel (random forests). Boosting fits learners sequentially, each correcting the last's residuals — this cuts bias from weak learners.
Convex vs non-convex? (ch11)
Classic ML losses (linear/logistic regression, SVM, softmax) are convex, so SGD finds the global optimum. Deep nets are non-convex, yet SGD still works: over-parameterization, good init, and the implicit regularization of SGD steer it toward flat, generalizing minima.
Frequentist vs Bayesian in one line? (ch3)
Frequentist: randomness is in the data, $\theta$ is a fixed unknown (MLE + confidence intervals). Bayesian: randomness is in your belief, $\theta$ gets a prior and you report a posterior (MAP/posterior-mean + credible intervals).
⚠ Clears up — what hesitation means
If any answer above is not instant, do not just peek at it and move on — that hesitation is the entire signal these notes exist to give you. Stop and reread the linked chapter (the number in parentheses) until the one-liner falls out automatically; a half-remembered answer in your head is a chapter you don't yet own.
◆ Interview probe — "derive that"
Take "why is L1 sparse" one level down. Minimizing $\tfrac12(\theta-z)^2+\lambda|\theta|$ (the 1-D proximal step) gives the soft-threshold $\hat\theta=\operatorname{sign}(z)\max(|z|-\lambda,0)$, which is exactly 0 whenever $|z|\le\lambda$. The L2 analog $\tfrac12(\theta-z)^2+\lambda\theta^2$ gives $\hat\theta=z/(1+2\lambda)$, which is 0 only when $z=0$ — so L1 zeroes a whole interval, L2 only scales.
◆ Interview probe — "show the gradient"
Push softmax+cross-entropy. With logits $z$, $p=\operatorname{softmax}(z)$, and one-hot $y$, the loss $-\log p_{\text{true}}$ has gradient $\partial\mathcal{L}/\partial z=p-y$. That single clean residual is why it composes through backprop and why the objective is convex in $z$ — and it's the same $\hat p-y$ form as logistic regression and linear regression under their matching exponential-family likelihoods (ch7, ch8).
◆ Interview probe — "what if the assumption breaks?"
Push generative vs discriminative. If the class-conditional Gaussian assumption of GDA is wrong, its small-data efficiency advantage evaporates and its asymptotic error is worse than logistic regression — which is why, with enough data, the discriminative model that makes fewer assumptions usually wins. The general rule: a generative model is a bet on $p(x\mid y)$, and you lose the bet when that density is misspecified.
Remember These are recognition drills, not lookups: an instant answer means you own the chapter, and any hesitation is your cue to reread the chapter in parentheses.
Tricky interview questions 12
Explain the bias–variance tradeoff in one breath.
Test error = bias$^2$ (too-simple → underfit) + variance (too-sensitive to the sample → overfit) + irreducible noise. Raising complexity lowers bias but raises variance, so you tune to minimize the sum. Trap: claiming you can minimize both at once for a fixed model class — only more data lowers variance without raising bias.
What is overfitting and how do you spot it?
Fitting noise in the training set: low train error, high validation error. The signature is the gap, not high train accuracy alone. Fix with more data, regularization (L1/L2, dropout), simpler models, early stopping. Trap: diagnosing it from training accuracy alone instead of the train–val gap.
L1 vs L2 — and why does L1 give sparsity?
L1 penalizes $\sum|w|$ and zeros weights (feature selection); L2 penalizes $\sum w^2$ and shrinks smoothly. Geometrically L1's diamond has corners on the axes; analytically its subgradient is a constant $\pm\lambda$ near 0, while L2's gradient $2\lambda\theta$ vanishes. Trap: saying L2 also zeros weights — its round constraint almost never produces exact zeros.
Gradient descent: batch vs mini-batch vs SGD — which do you prefer?
Step against the gradient. Full-batch is stable but slow/memory-heavy; SGD (one example) is noisy and fast and escapes shallow minima; mini-batch is the default — balances stability and speed and uses GPU vectorization. Trap: saying full-batch is best because its gradient is "exact" — it's slow and its lack of noise can stick in sharp minima.
Precision vs recall — when optimize each?
Precision = TP/(TP+FP); recall = TP/(TP+FN). Optimize precision when false positives are costly (spam); optimize recall when false negatives are costly (cancer, fraud). Trap: swapping the denominators, or claiming you can max both — they trade off as you slide the threshold.
ROC-AUC vs PR-AUC — which under imbalance?
ROC-AUC ranks across thresholds, fine when balanced. Under heavy imbalance the huge true-negative count keeps FPR low and ROC looks optimistic, so use PR-AUC (or F1) which focuses on the rare positive. Trap: trusting a near-1.0 ROC-AUC on a 1%-positive problem — the model can still be useless on the minority class.
Bagging vs boosting?
Bagging = parallel models on bootstraps, averaged → cuts variance (random forest). Boosting = sequential, each fixes the prior's residuals → cuts bias (GBM, AdaBoost). Trap: saying both fight overfitting equally — boosting can overfit if over-iterated; bagging rarely does.
Random forest vs gradient boosting — when each?
RF builds deep decorrelated trees independently and averages (variance reduction, parallel, robust). GBM builds shallow trees sequentially on residuals (bias reduction, higher accuracy, more tuning). RF = robust baseline; GBM = top accuracy when you can tune. Trap: calling GBMs "just parallel like RF" — boosting is inherently sequential.
Vanishing gradients — cause and fixes?
Backprop multiplies many small derivatives, so early-layer gradients shrink to ~0 and stall (worse with saturating sigmoid/tanh). Fix: ReLU-family, He/Xavier init, batch/layer norm, residual connections, gating (LSTM/GRU). Trap: confusing it with exploding gradients, or thinking a smaller learning rate fixes it — the signal itself is decaying, not the step size.
What is data leakage and how do you prevent it?
Information unavailable at prediction time (future data, the target, test-set stats) sneaks into training, inflating offline metrics that collapse in production. Split first; fit all preprocessing on train only; drop target-derived/future features. Trap: scaling, imputing, or selecting features on the full dataset before the split — it contaminates the test set even when the split looks clean.
Generative vs discriminative — and the classic gotcha?
Discriminative learns $P(y\mid x)$ directly (logistic regression, SVMs, most nets); generative learns the joint $P(x,y)$ via $P(x\mid y)P(y)$ (Naive Bayes, GMMs) and can generate data / handle missing inputs. Trap: calling Naive Bayes discriminative or logistic regression generative — NB models the joint, LR models $P(y\mid x)$, even though they're a known pair.
When standardize features, and which models care?
Scale for distance- and gradient-based models: KNN, k-means, SVM, PCA, neural nets, regularized linear — so no feature dominates by units. Tree models (RF, GBM) are scale-invariant. Trap: scaling for trees (pointless), or computing the scaler on the full dataset instead of fitting on train only (leakage).
That's the core
Probability & estimators → the supervised models → optimization → learning theory → unsupervised → trees & ensembles → RL → evaluation, in CS229 order — each page with a hook, a figure, and a bank of tricky interview questions. Drill them with the flashcards.