Mock interviews

Approach

1. Sync version: queue.Queue for unvisited URLs, set for visited. Dequeue, fetch HTML, parse links, filter same-domain + strip fragment, push unseen.
2. Concurrent: ThreadPoolExecutor(max_workers=N). Submit fetch tasks; on completion, parse + enqueue new URLs. Visited set guarded by lock.
3. Termination: maintain in-flight counter; when queue empty AND no in-flight tasks, done.

Follow-up answers

• Politeness: per-host token bucket; cap concurrent requests per host.
• robots.txt: fetch and cache per host; respect Disallow.
• Distributed: shard URLs by hash(URL) % N workers; central dedup via Redis or Bloom filter; coordinator detects "done."
• Threads vs processes: GIL is released on socket reads → threads are fine. Processes only if HTML parsing dominates.

Base structure

Hashmap key → list of (timestamp, value) sorted by timestamp. set: append (assume monotonic) or insort. get: binary search for largest ts ≤ target.

Extensions

• Per-key locks: stripe-lock on hash(key) % N — much higher concurrency than global mutex on hot keys.
• Persistence: append-only log (write-ahead) + periodic snapshot. Recover by replay since last snapshot.
• TTL: store (timestamp, value, expiry); lazy eviction on read; periodic background sweep.

Edge cases interviewer expects you to mention

• Out-of-order timestamps. Concurrent updates with same timestamp. Memory growth without eviction. Crash mid-write (use fsync semantics).

Approach

1. Parse formula to find dependencies. Build a dependency graph.
2. Cycle detection: DFS with 3-color marking (white/gray/black). Gray-on-gray = cycle.
3. Eager evaluation: on setCell, topological-sort affected cells; recompute in order. getCell = O(1) lookup.
4. Lazy alternative: cache evaluations; invalidate on dependency change.

Common follow-ups

• Concurrent reads/writes (RWLock). Memory cost of dependency graph. Streaming partial updates.

Structure

Hashmap (key → node) + doubly-linked list (head = most recently used, tail = least recently used).

• get: hashmap lookup; move node to head; return value.
• put: if key exists, update value + move to head. Else create node, push to head, hashmap insert. If over capacity, remove tail node + remove from hashmap.

Pitfalls

• Forgetting to remove from hashmap when evicting tail. Sentinel head/tail nodes simplify edge cases. Single mutex around the whole struct for thread safety.

LeetCode 146

Approach

1. Hashmap counts: Counter(nums).
2. Min-heap of size K: push (count, num); if size > K, pop. End: heap has K largest.
3. Alt: bucket sort by count → O(N) but uses more memory.
4. Alt: quickselect on counts → O(N) average.

LeetCode 347

Code skeleton

class MHA(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.h = n_heads; self.d_h = d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)
        self.out = nn.Linear(d_model, d_model)

    def forward(self, x, causal=True, pad_mask=None):
        B, T, _ = x.shape
        qkv = self.qkv(x).reshape(B, T, 3, self.h, self.d_h).permute(2, 0, 3, 1, 4)
        q, k, v = qkv[0], qkv[1], qkv[2]   # (B, H, T, d_h)
        scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.d_h)
        if causal:
            mask = torch.triu(torch.ones(T, T, device=x.device), 1).bool()
            scores = scores.masked_fill(mask, float('-inf'))
        if pad_mask is not None:
            scores = scores.masked_fill(pad_mask[:, None, None, :], float('-inf'))
        attn = F.softmax(scores, dim=-1)
        out = (attn @ v).transpose(1, 2).reshape(B, T, -1)
        return self.out(out)

What interviewers check

• Did you scale by √d_head? (Why: prevents softmax saturation.)
• Did you mask before softmax? (Not after.)
• Shape gymnastics correct? Comment shapes.
• Bonus: extend to GQA. Extend to KV cache for autoregressive decode.

Algorithm

1. Initialize vocabulary as bytes (256 entries).
2. Pre-tokenize corpus (split on whitespace + simple regex).
3. Count adjacent pair frequencies in current word splits.
4. Find most frequent pair. Add merged token to vocab. Apply merge to all words.
5. Repeat until target vocab size.

Encode

Apply learned merges in order (priority by merge index). Greedy left-to-right.

Common pitfalls interviewer probes

• Whitespace handling: GPT-2 BPE includes leading space as part of token.
• Byte fallback: unknown characters → emit raw bytes; vocab guarantees byte coverage.
• Merge order matters at encode time.

Code

def sample(logits, temp=1.0, top_k=None, top_p=None):
    logits = logits / max(temp, 1e-6)
    if top_k:
        idx = np.argpartition(logits, -top_k)[-top_k:]
        mask = np.full_like(logits, -1e9); mask[idx] = logits[idx]
        logits = mask
    # numerically stable softmax
    logits -= logits.max()
    probs = np.exp(logits) / np.exp(logits).sum()
    if top_p:
        order = np.argsort(-probs)   # stable descending
        cum = np.cumsum(probs[order])
        cutoff = np.searchsorted(cum, top_p) + 1
        keep = order[:cutoff]
        new = np.zeros_like(probs); new[keep] = probs[keep]
        probs = new / new.sum()
    return np.random.choice(len(probs), p=probs)

What gets you graded down

• Forgetting logits -= logits.max() for stability.
• Off-by-one in nucleus cutoff — must keep at least one token whose cum ≥ p.
• Not handling temperature = 0 (greedy degenerate case).

Algorithm

1. k-means++ init: pick first centroid uniformly random; each subsequent centroid sampled with probability ∝ D(x)² where D(x) is distance to nearest existing centroid.
2. Assign: each point to nearest centroid (vectorized with numpy broadcast).
3. Update: centroid = mean of assigned points. Empty cluster → reinitialize via k-means++ or to a random point.
4. Stop: when assignments stop changing OR centroid movement < ε OR max iterations.

Follow-ups

• Choose K: elbow method (within-cluster SSE vs k), silhouette score, gap statistic.
• Scale to 1B points: mini-batch k-means; or k-means on a subsample then assign rest.
• High-d data: PCA first; or use spherical k-means for unit-norm vectors.

Algorithm sketch

1. Draft generates K tokens, recording draft probabilities at each step.
2. Target runs ONE forward pass over (prefix + drafted tokens) → next-token distributions at every drafted position.
3. For each drafted token t: accept with probability min(1, p_target(t) / p_draft(t)).
4. On reject: sample one corrected token from (p_target − p_draft)_+ / Z; stop.
5. If all K accepted: sample one bonus token from target's last distribution.

Why exact

Marginal distribution of each emitted token = p_d(t) · accept(t) + (1 − E[accept]) · residual(t). Algebra works out to p_t(t). Each emitted token is exactly distributed as the target.

Funnel

1. Candidate generation (1B → 1k): two-tower (user/item embeddings) + ANN over item index. Multiple sources merged: collab, fresh, subscriptions, trending.
2. Ranking (1k → 100): cross-encoder transformer or DLRM with user history (DIN-style attention) + multi-task heads (pCTR, pWatchtime, pLike, pDislike) via MMoE.
3. Re-rank (100 → 10): final scoring (weighted heads), diversity (MMR / DPP), freshness boost, business rules.

Training infra

• Embedding tables (TBs) sharded via TorchRec. Streaming Kafka → Flink → trainer for freshness. Daily full retrain + hourly incremental.

Eval

• Offline: AUC, NDCG, recall@k, calibration. Online: A/B with primary watchtime, guardrails on dislikes, diversity, abandonment.

Gotchas to mention

• Position bias → estimate via shallow tower or PAL.
• Feedback loop → mandatory exploration slots.
• Clickbait → train pWatchtime + pSatisfaction, not pCTR.
• Train/serve skew → same feature pipeline, point-in-time correctness.

Full deep dive: ML system design.

Architecture

• Disaggregated serving: prefill pool (compute-bound) + decode pool (bandwidth-bound). KV transferred over RDMA.
• Prefix cache: hash on prompt prefix tokens; share KV across requests with common system prompt / few-shot.
• Continuous batching: per-iteration scheduling. New requests join, finished requests leave.
• Tensor parallel within node (TP=8 for 70B+); pipeline parallel across nodes only for very large.
• Speculative decoding with EAGLE-style draft head.
• Quantization: FP8 weights+activations; INT4 weights for cost-tier variants.
• Routing: model router (cheap classifier on prompt) selects mini / full / reasoning model based on query complexity.

Eval / monitoring

• Offline evals (MMLU, HumanEval, MATH, internal). Online: thumbs feedback, conversation length, retry rate, A/B on model versions.
• Monitor TTFT / ITL distributions, KV cache hit rate, GPU util, queue depth, refusal rate, output toxicity.

Gotchas

• Head-of-line blocking from long-context requests → priority queues, separate pools.
• Autoscaling lag (GPU spin-up takes minutes) → predictive scaling.
• Safety filtering pre/post adds latency.

Full deep dive: LLM inference.

Pipeline

1. Data ingest: prompts (real, red-team, synthetic). Dedupe + PII filter.
2. Generation pool: base model produces K candidates per prompt; self-critique via constitutional principles; revisions.
3. Preference labeling: AI judge (stronger model) ranks pairs against constitution. Subset labeled by humans (gold).
4. Training: SFT on revisions; then DPO/KTO/PPO on preferences. Iterate.
5. Evaluation: capabilities (MMLU, MATH, HumanEval), safety (HarmBench, XSTest), instruction following (IFEval), preference winrate vs prior model.
6. Regression gate: blocked if N safety/capability metrics regress > ε.
7. Canary rollout: 1% → 10% → 100% with online monitoring.

Pitfalls to mention

• Reward hacking, mode collapse in DPO, distribution shift between SFT data and RL prompts, eval contamination, safety-helpfulness trade-off.

Approach

1. Token bucket per key, stored in Redis with Lua atomic update (refill based on elapsed time × rate, capped at capacity).
2. Higher scale: shard Redis by hash(key) % N.
3. Even higher scale: local approximate counters per service, periodic sync to Redis. Trades precision for throughput.
4. Leaky bucket alternative: smoother but no burst tolerance.
5. Sliding window log: most precise, highest memory.

Gotchas

• Hot keys (single user / DDoS) — shard within key (key + random salt buckets).
• Clock skew across services → use Redis time as authority.
• Failure mode: if Redis is down, fail open (allow) or fail closed (deny)? Usually fail open with degraded service banner.

STAR template

• Situation: name the launch, the team, the deadline, the pressure.
• Task: what specifically YOU owned (don't say "we").
• Action: the data you saw or the test that flagged the concern. The decision to delay. Who you had to convince. The communication path (engineering manager, PM, ultimately the partner team / leadership).
• Result: the time impact + the bug or risk you avoided. Quantify both.

What lands

• You held a position despite shipping pressure. You had data. You took responsibility for the call (not "I escalated and let leadership decide").
• You communicated transparently with stakeholders rather than blocking silently.
• The fix didn't come at the cost of demoralizing the team; you made the path forward concrete.

Three honest framings — pick one

1. Scope ceiling. "I've been Staff on [my area] for X years. To go Sr Staff at my current company requires more ladder time. I'd rather earn that level somewhere I'm pushed harder on a different problem."
2. Mission shift. "What I've been doing at scale is well-trodden ground. I want to be on the LLM/AGI frontier where the hard problems are unsolved."
3. Founder energy. "Want a smaller-org bet. Pre-IPO equity at a frontier lab is a bet I want to take while I can."

Don't say

• "Meta is slowing down" / "TBD Lab took all the fun roles" / "My manager is bad" / "Burned out." All read as flight-risk.

Full playbook: Behavioral page.

Structure

• Pick a real one. Quantified cost. "Missed launch by 2 weeks costing X% of quarterly OKR." Or "regression that affected Y% of users for Z hours."
• Take responsibility cleanly. "I underestimated X" — not "the team didn't have time."
• Specific lesson. Name the concrete thing you changed in your process / system afterwards.
• Show the next instance went better. "When the same kind of decision came up six months later, I [did the new thing]."

Anti-patterns

• "My biggest weakness is I work too hard." Cliché. Penalty.
• Failure that was actually someone else's fault.
• No quantified impact — "it was a learning opportunity."

Reference points (use 2)

• Constitutional AI as a path to scalable alignment vs RLHF's labeler bottleneck.
• RSP / ASL framework — explicit capability thresholds that gate safety practices.
• Interpretability investment (Templeton 2024 Scaling Monosemanticity, Olsson induction heads).
• Mech interp as a plausible certification path for safety claims (vs purely behavioral evals which Sleeper Agents showed can be fooled).

Tie to your work

"I've spent X years building [your area — e.g. ranking systems where calibrated probabilities matter]. The interp work feels like the same kind of building-up-trust-via-measurement problem at a different scale, and I want to be on that side of it."

The EMA m_t = β m_{t−1} + (1−β) g_t with m_0 = 0 unrolls to:

m_t = (1−β) Σ_{i=1}^{t} β^(t−i) · g_i

Under stationary g: E[m_t] = (1−β^t) · E[g]. Biased toward 0 by factor (1−β^t). Divide by it: m̂_t = m_t / (1−β^t) is unbiased.

Why it matters: at step 1 with β=0.9, raw m has 10% the magnitude it should. First update is 10× too small without correction.

Geometric: L2 ball is a smooth circle/sphere. Loss contour first touches at a generic point — no coordinate exactly 0. L1 ball is a diamond/hyperoctahedron with corners on the axes. Loss contour most often touches at a corner — corners have all-but-one coordinate = 0 → sparsity.

Gradient: L2 gradient is 2λθ — proportional to weight, pulls toward 0 multiplicatively (never reaches it). L1 subgradient is λ · sign(θ) — constant pull regardless of magnitude. Soft-threshold operator sign(θ) · max(|θ| − λ, 0) drives small weights to exactly 0.

Bayes: L2 = Gaussian prior MAP. L1 = Laplace prior MAP (Laplace has heavier mass at 0).

• Probabilistic correctness: CE is the negative log-likelihood under Bernoulli/Categorical noise; MSE assumes Gaussian noise (wrong for discrete labels).
• Gradient signal: CE gradient w.r.t. pre-softmax logits is p̂ − y — clean, bounded, doesn't vanish. MSE through softmax includes σ'(z), which vanishes when softmax saturates → slow learning.
• Calibration: CE produces calibrated probabilities. MSE-trained classifiers are systematically miscalibrated.

Chinchilla (Hoffmann 2022): for fixed training compute C ≈ 6ND, the loss-minimizing model has D ≈ 20·N tokens per parameter. Pre-Chinchilla models (GPT-3, Gopher) were undertrained.

Why 2026 violates this: training compute is one-shot; inference compute is amortized over years of serving. Llama 3 8B trained on 15T tokens (1875 tokens/param) is way past Chinchilla optimal — but the smaller model is much cheaper to serve, so total system cost is lower over the model's lifetime.

The compute-optimal frontier shifts when you include inference cost in the objective.

RoPE is a non-linear transformation depending on token position — it cannot be absorbed into Q's projection because the rotation depends on which token you're looking at, not the projection weights.

So MLA splits Q and K into two parts:

• Q^C, K^C: content path. Compressed via low-rank latent c; no RoPE; up-projection absorbed into Q.
• Q^R, K^R: rotary path. Small dim (e.g., 64); RoPE applied; K^R shared across all heads (MQA-style).

Final attention score = Q^C K^C + Q^R K^R. Cache stores only c + small K^R. ~7% of MHA cache size with MHA-equivalent quality.

Llama 3 70B uses GQA: 8 KV heads, d_head = 128, 80 layers, BF16.

• Per token per layer: 2 · 8 · 128 · 2 = 4,096 bytes (the leading 2 = K + V)
• Per token (all layers): 4096 · 80 = 327,680 ≈ 320 KB
• At 128k tokens: 320 KB · 131,072 ≈ 40 GB per request
• If MHA (64 KV heads): 8× → ~320 GB per request — totally unservable. That's why GQA exists.
• MLA (latent dim ~512): ~3 GB at the same context — >10× smaller.

• Selection bias / exposure bias — train log only contains items the old model showed; new model's recommendations are out-of-distribution.
• Train/serve skew — different feature pipeline at training vs serving.
• Miscalibration — AUC measures ranking, not calibrated probability; downstream uses (auctions, thresholds) suffer.
• Distribution shift — historical traffic mix doesn't match production.
• Novelty effect — short-term spike from "new content" without sustained engagement.
• Position bias — model fits old position-weighted clicks; new ranking changes positions, breaking the assumption.

log Σ exp(x_i) overflows if any x_i is large (e.g., 1000 → exp explodes). Subtract the max:

log Σ exp(x_i) = m + log Σ exp(x_i − m)
where m = max(x_i)

Now exp arguments are ≤ 0, can't overflow. Used everywhere: softmax, partition functions, anything that's "log of a sum of exponentials." Forgetting this is the #1 source of NaN losses in custom kernels.

• In-batch: cheap (no extra forward); popularity-biased (high-frequency items appear more often as negatives, get pushed away too aggressively); fix with logQ correction (Yi 2019): subtract log(sample_freq) from logits.
• Hard negatives (top non-positive from ANN over corpus): faster per-example learning; risk of false negatives (true positives that look similar); risk of optimization instability.
• Mixed Negative Sampling (MNS): combine in-batch (popularity-biased, cheap) + uniform-random (cheap, less biased) + hard (expensive, fast learning). Best of all worlds.
• Cross-batch: queue past embeddings as additional negatives (MoCo-style). Larger negative pool with constant memory.

• Maintain stack of active frames. For each sample, find longest common prefix with previous stack. Past the prefix on old stack = exits (innermost-first); past it on new stack = enters (outermost-first).
• Frame identity: (name, depth) works in simple cases; for recursion give each invocation a unique id at enter.
• First sample: everything enters. Streaming variant: bounded fidelity — emit at each sample with no lookahead.
• Source: Glassdoor; LinkJob Anthropic coding bank Q1.

• ThreadPoolExecutor, worker pulls from thread-safe queue, fetches with requests, parses, dedupes via set+Lock.
• Asyncio strictly better: I/O-bound, GIL forces threads to serialize on bytecode even when blocked on sockets. asyncio.gather with thousands of coroutines on one thread sidesteps the GIL.
• Multiprocessing wrong here: network-bound, IPC dominates. CPU-bound parsing would change the answer.
• Memory contention: visited-set is the real hotspot — shard by URL hash for sharded sets.

• Bugs to find: (i) UTF-8 byte-boundary splits making undecodable strings; (ii) unknown chars not falling through to <unk> or byte-fallback; (iii) greedy match missing a longer downstream token; (iv) detokenize losing leading-space metadata (BPE ▁).
• Discuss BPE vs WordPiece vs SentencePiece. Why GPT-4o went to ~200k vocab.
• Anthropic wants production-shaped: pure functions, explicit error types, no hidden mutable state.

• Mode: each machine builds local Counter. Tree reduction — round 1: nodes 0..4 → 5..9 merge; round 2: 5,6 → 7,8. Root has global counter. O(unique × log N) messages.
• Median: binary-search the value. Each machine reports count(x ≤ v); root sums and adjusts. O(log(range) × N) — far cheaper than gather.
• barrier synchronizes rounds; without it, send/recv reordering races.
• Anthropic looks for bandwidth-budget intuition, not perfect MPI.

• Bugs: list not doubly-linked (O(n) move-to-front); stale dict refs; eviction race with put; missing head/tail sentinels.
• Durability: WAL (log → disk → ack) vs synchronous snapshot. fsync boundaries; recover by replay; cap log size via periodic snapshot.
• Concurrency: RwLock isn't a free win — writer starvation. Discuss.

• Split words into chars + EOW marker. Count adjacent pair frequencies; pop max; merge everywhere; push merged symbol; repeat N times.
• Naive O(N × V × L) is fine for the interview but mention the smart version: inverted index pair → set of word ids so each merge only touches affected words (HF tokenizers).
• Byte-level BPE (GPT-2): no <unk> ever. Tokenizer trained on code splits Python idioms differently than English.
• Probe: "frequent pair that wouldn't merge well?" — pairs spanning punctuation (weird inter-word tokens).

• Continuous batching (vLLM): when any request finishes, swap in a new one — >85% utilization.
• PagedAttention: KV as fixed-size blocks indexed by page table; no fragmentation at mixed lengths.
• Prefix caching: huge for agentic workloads sharing 200K toolset prefixes.
• Chunked prefill: split long prefills across decode steps to avoid starving decoders.
• Priority queueing + preemption (chat vs batch streams).
• Speculative decoding (EAGLE-3 / MTP): 2-3× speedup at low concurrency; turn off above ~32 concurrent because it burns FLOPs that could serve other requests.
• Anthropic wants you to talk Goodput (useful tokens delivered within SLA), not raw throughput.

1. Hardware: flaky node? All-reduce checksum; NaN grads localized to a rank. Failing GPUs flip silent bits in BF16.
2. Data: pathological shard (all-zero, very long docs, leaked test). Hash last 1K examples vs known-good.
3. LR/Adam: second moment can collapse on repeated samples → effective LR explodes. Z-loss on logits, grad clip 1.0, AdamW eps ≥ 1e-8.
4. Numerics: softmax overflow in FP16/FP8; attention logits exceeding format max; FP8 scale drift.
5. Schedule: cosine LR transitions, weight-decay phase change.
6. Mitigate: skip bad microbatches, rewind to recent ckpt, lower LR. PaLM rewound 100 steps and skipped 200-500 batches.

• CLT: SAE-like decomposition where features can read from any earlier layer and write to any later layer. Replaces MLP blocks in a substitute model.
• Steps: train CLTs against frozen target; at inference, freeze attention pattern from real model (attention is hard to interpret directly), run substitute MLPs; compute direct attribution from feature i at layer L to feature j at layer L′ as the gradient of j's preactivation w.r.t. i's activation through linear paths; prune low-attribution edges.
• Example: "Dallas is in" → "Texas" lights up Dallas → city-in-Texas → state-name sequentially.
• Limitations: CLTs only explain MLPs (attention is fixed); completeness is unmeasurable; perturbation tests are required to confirm hypotheses.
• Anthropic specifically probes whether you understand the graph is over substitute features — the price of monosemanticity.

• MLA compresses K,V into shared latent c_KV ∈ R^{d_c} (d_c ≈ 512 vs 16384). At inference cache only c_KV, up-project on the fly.
• RoPE rotates position-dependently. You can't apply RoPE pre-compression (entangles position with shared latent) and can't apply it post-up-projection cheaply.
• Decoupled RoPE: split key into [k_no_rope; k_rope]. k_no_rope goes through the latent path; small k_rope is separately stored, rotated, shared across heads.
• V4 abandons MLA for CSA + HCA: chunked local windows + hierarchical "summary" tokens. KV cache 10% of V3.2; FLOPs/token 27%. Loses some needle-in-haystack unless retrieval supplements training.

• Eval registry: versioned defs (dataset hash, prompt template, grader spec, seed).
• Sandbox executor: rate-limit handling, retries, provenance.
• Auto-grader bank: string-match, regex, code-exec (containerized), LLM-judge with calibrated panel + held-out human labels for drift.
• Caching keyed on (model_id, prompt_hash, seed).
• Statistics: Wilson CIs; auto-flag >2σ regression from parent ckpt.
• Contamination check: n-gram match against pretraining data.
• LLM-judge must be a separate model family — can't use Claude to judge Claude.
• Probe: "what if the grader is drifting?" — frozen human-judged ref set; track grader-vs-human agreement; rebuild grader when agreement drops.

• STAR. They want: (i) you actually pushed back, not "I agreed"; (ii) a concrete proposal (what data would change your mind, what tests); (iii) calibration about when shipping is right.
• Structure: name a feature, specific risk (e.g., red-team gap on multilingual jailbreak), cost (delay days, revenue), proposal (delay 2 weeks, add 5 specific evals, quantitative bar), outcome, what you'd do differently.
• Probe: "what if you'd been wrong about the risk?" — don't get defensive; describe the experiment that would have falsified you.

1. Position embeddings initialized as torch.zeros instead of torch.randn × small std → can't break positional symmetry. Fix: nn.init.normal_(pos_embed, std=0.02).
2. Causal mask uses 0 instead of -inf in the upper triangle → softmax still attends to future. Fix: masked_fill_(upper_tri, float('-inf')) before softmax.
3. Missing loss.backward() between zero_grad and step → loss constant.
4. nn.Linear(d_model, n_heads × head_dim) with wrong out-features (d_model directly) — silent shape bug if head_dim × n_heads ≠ d_model.

Bonus: KV cache — list per layer; on new token only compute Q for new position, append new K,V row. Don't use it during training (caches stale grad-blocked tensors).

• kNN: dists = (X_test**2).sum(1)[:,None] + (X_train**2).sum(1)[None,:] - 2 * X_test @ X_train.T; preds = y_train[dists.argmin(1)].
• MLP: [Linear, ReLU, Linear] → embedding. Triplet loss max(0, d(a,p) - d(a,n) + α) or contrastive.
• Probe: "L2 after linear ≠ learned similarity?" — distance after W is (x-y)ᵀWᵀW(x-y), a Mahalanobis distance with PSD metric WᵀW. The network learns the metric.

• Token counter sidecar at serving tier (server-side tokenization — can't lie). Emits (user_id, model, n_in, n_out, ts) to Kafka.
• Streaming aggregator (Flink/Materialize) rolls to 1-min/1-hr/1-day in Redis.
• Quota service: Redis INCRBY + Lua check-and-charge; 429 over limit. Token bucket with refill.
• Billing: daily batch from S3 cold log; dedupe by (request_id, ts); price; invoice.
• Eventual consistency: enforcement = Redis; billing = durable log; they can diverge minutes; reconcile EOM.
• Fraud: gradient anomaly on per-user TPS.
• Tradeoff: strong-consistent SQL gives perfect billing but throttles ~10K TPS; eventually-consistent Redis gives 100× scale and overshoots limits by seconds.

• Forward: standard. Loss: L = -log p[y].
• Backward: dL/dz2 = p - one_hot(y) (softmax+CE shortcut). dW2 = a1.T @ dz2; db2 = dz2.sum(0). da1 = dz2 @ W2.T. dz1 = da1 * (z1 > 0) (ReLU subgrad). dW1 = x.T @ dz1; db1 = dz1.sum(0).
• Bugs interviewers plant: forgetting batch-mean of grads (dW explodes); wrong sign on log-softmax; int y vs one-hot indexing mismatch.
• Follow-up: derive without softmax+CE pairing — confirms you understand per-layer Jacobians.

• Dawid-Skene EM: jointly estimate per-annotator confusion matrices and latent labels.
• Flag bad annotators by test-retest self-agreement on duplicates; weight votes lower.
• Confident Learning (Northcutt): high-CE-on-high-confidence-pred = candidate errors; audit.
• Co-teaching: two networks, each selects low-loss samples for the other's update.
• Symmetric / generalized CE: bounded loss (1-p_y^q)/q robust to label noise (CE has unbounded grad on mislabels).
• 50% adversarial? Abandon supervised; self-supervised pretrain + tiny verified-label finetune.

• Tier routing: cheap classifier sends easy → small model (4o-mini class), reasoning → o-class, code → specialist. ~70% on cheap models cuts cost 5×.
• Speculative decoding default-on (EAGLE-3); off above 32 concurrent.
• Prefix caching across conversations: cache system prompt + last 10 turns; hit rate >50%.
• Region-affinity: same conversation routes to same shard for 5-min stickiness — KV locality.
• NVFP4 inference (3.5× memory, <1% quality loss); FP8 on H100 fleet.
• KV tiering: HBM → CXL DRAM → NVMe with prefetch for >200K context.
• Track $/successful-conversation, not $/token.
• L5 differentiator: reason about operational costs (oncall, deploys, hotfixes).

• Per-key list[(version, value)]; get(k, v) = binary search.
• MVCC: txn sees snapshot at start; writes tagged with txn_id. On commit: assign next global version; check no other txn committed a higher version for any key in the read set.
• WAL: append (txn_id, [(k,v)…]) before applying. On crash, replay.
• Snapshot isolation default; serializable needs SSI (RW conflict graph).
• Phantom reads on range scans: track range predicates, not just keys.

• Vanilla writes full n×n to HBM, three round-trips. FA computes block-wise in SRAM with online softmax (running max + sum) so partial softmaxes combine without re-reading.
• FA2: reverse-pass tile reuse, warp-level partition.
• FA3 (Hopper): overlap matmul (Tensor Cores) with softmax (CUDA cores) via warp specialization; TMA for async copies.
• FA4 (Blackwell): WGMMA → TCGEN05 ops in tensor memory; exp() via polynomial on FMA units (SFU now the bottleneck); lazy softmax rescaling skips rescale when shift < stability ε; explicit multi-stage multi-buffer pipeline.
• B200 BF16 ~1605 TFLOPS, 71% util, 1.1-1.3× cuDNN.
• FA is memory-IO, not algorithmic — still O(n²) FLOPs.

• Standard KD: student trains on teacher trajectories — off-policy. Student visits states the teacher never saw → compounding error.
• RL: student samples own trajectories (right distribution) but sparse scalar reward → wasteful credit assignment.
• OPD: student rollouts + dense per-token KL to teacher's distribution. Right state distribution AND dense supervision.
• 5-20× fewer compute steps than RL on reasoning (TML, Qwen3-OPD).
• Preconditions (Rethinking OPD 2026): (i) teacher/student share compatible thinking patterns; (ii) teacher offers new capability beyond self-play.
• Failure: if student already as capable as teacher, OPD wastes compute vs RLVR.

• Reduction kernels pick different tree shapes per batch size (autotuner). FP add isn't associative → bitwise-different outputs.
• Matters because: (1) reproducible RL — different logits → broken importance ratios; (2) deterministic inference for safety/debug; (3) speculative-decoding verifier needs same logits regardless of co-batch.
• Fix: replace reductions with fixed-tree impls (enforce block size + reduction order). 5-15% throughput cost for bitwise determinism.
• Not "deterministic GPU" — kernel still races; all races produce same result because reduction shape is fixed.

• Researchers iterate on the RL loop itself (PPO/GRPO/DPO variants); they need gradient + step control, not a black-box trainer.
• LoRA-only: tiny per-user adapters (~50-200M params) — server shares one base model across users; each user holds LoRA. Economically viable.
• Stateless from user perspective: pure-function primitives over server-held ckpt; client code is portable.
• Tradeoffs: no full SFT-from-scratch; can't change base; LoRA caps how much new behavior installs (~10% relative gain).
• Comparison: Anthropic Workbench (prompt-only) / OpenAI fine-tuning (managed). Tinker = "I want PPO on Qwen-72B without owning the cluster."

• Replaces "user turn → assistant turn" with continuous interleaved streams. Every 200ms: audio+video tokens in, audio+text tokens out.
• dMel: direct mel spectrogram discretization via lightweight embed — no Whisper-class encoder, no encoder latency.
• 40×40 image patches via hMLP, co-trained from scratch with the transformer. Encoder-free early fusion.
• Multi-stream output head: emits 200ms audio chunk + optional text; if user speaks, downweight audio stream.
• Native time as a positional modality (so "remind me in 5 min" works).
• Persistent GPU memory sessions (TML's SGLang upstream): keep KV hot across micro-turns; skip 50ms re-alloc per tick.
• TML-Interaction-Small (276B-A12B MoE): <0.4s turn-take on FD-bench vs Gemini 3.1 flash-live 0.57s.
• Failure: model can talk over user → backchannel token type emitted when detecting active user speech.

• Muon: replace AdamW's element-wise scaling with 5-iter Newton-Schulz orthogonalization of the momentum. Update W ← W - lr × orthogonalize(M).
• Empirically, orthogonalized updates have bounded spectral norm regardless of which weight matrix — one LR works across all hidden layers (AdamW needs per-layer tuning). 1.5-2× wall-clock at same loss.
• Manifold Muon: enforce weights themselves lie on a well-conditioned manifold (bounded singular values). Slow inner loop; Buchanan's ADMM variant gives 2× wall-clock.
• Bounded condition number → free numerical stability under NVFP4; update directions naturally aligned with preconditioner.
• 100B from scratch? Honest: only Moonshot/Kimi has reported >70B Muon. Others test at 1-7B. Risk: NS iterations interact poorly with FP8 — intermediate Gram matrices need wider range.

• Combinatorial diversity: C(256,8) ≈ 4e9 vs C(8,2)=28. Each token specializes across a much wider feature subspace at the same total params.
• Shared expert captures generic knowledge so routed experts aren't burning a slot on it.
• Aux-loss-free balancing: per-expert bias offsets to router logits, tuned by observed utilization — avoids gradient interference from LB losses.
• ~5× more diverse routing + ~30% less LB tax than Mixtral.
• Cost: 256-expert layers need careful all-to-all kernels (EPLB, DeepEP). On <8 GPUs, fine-grained MoE doesn't fit one node — comms dominate.

• V4-Pro 1.6T/49B activated; V4-Flash 284B/13B; both 1M context.
• MLA dropped → CSA + HCA hybrid: chunked sliding (windows ~4096) interleaved with hierarchical compressed attention (a few hundred global summary tokens). MLA's compression ratio would have needed to grow superlinearly at 1M.
• Per-token FLOPs at 1M context dropped 73%; KV cache occupancy 10% of V3.2.
• Native MTP heads stay (4 prediction heads, trained jointly) → 1.8× speculative-decoding free at serve.
• Still H800 fleet — no Blackwell — so kernel-level efficiencies (custom comms, FP8) carry the day.
• Signals: arch space not solved (V3's "MLA = answer" was wrong); long-context economics dominated by KV cache, not FLOPs; MoE granularity still rising; "we're going dense to keep things simple" is not on the efficient frontier.

• Pad text to a multiple of 8 (or 64 on Hopper/Blackwell), not literal max — tensor-core friendly. Build attention_mask.
• Resize with F.resize bilinear, antialias=True, then normalize. Aspect-ratio: short-side resize + center crop OR letterbox with padding mask. Gemini uses patch-and-pack with valid_patches mask.
• GPU resize: torchvision.transforms.v2; large batches → DALI / webdataset for overlap.
• 10% corrupted images? Emit "dropped" boolean mask, skip downstream.

1. Distill with Gemini Pro as teacher. On-policy distillation (student rollouts + dense teacher KL). 3-5× compression.
2. Structured pruning (channel magnitude, RL selector like NetAdapt). 1.5-2× more.
3. Quantize: INT4 weights + INT8 activations via GPTQ/AWQ (activation-aware: scale by output-channel activation magnitude).
4. QAD: co-train distill + quant noise to recover PTQ losses.
5. Eval: multilingual + reasoning + code. Per-task gating: if reasoning loses 8%, route to a larger ckpt.
6. Deploy: ml-package; INT8 KV cache halves on-device memory.
7. Pure PTQ INT8 = 4×, not 10×. Aggressive transformer pruning cliffs at 30-50%.

• Pre-gen prompt classifier: multimodal embed (SigLIP-class) → harm head. Block at front.
• Constrained generation: suppress banned tokens; partial-output classifier every K tokens; abort over threshold.
• Post-gen VLM classifier on full output (text+image).
• Roleplay-framed harm: meta-classifier on roleplay intent; train on FORBIDDEN-roleplay.
• Calibration: FP rate matters. Per-category P/R on holdout; drift monitoring.
• Cross-language: multilingual encoder + per-lang head — low-resource attack pattern.
• Red-team loop: weekly retrain on new attacks.

• FP8: ~20 TB weights, ~40 TB optim state. Must shard heavily.
• 3D parallelism: TP=8 (intra-node, NVLink) × PP=~8 stages (cross-node, IB) × DP=rest.
• ZeRO-3/FSDP inside each DP group.
• MoE? Expert parallel across nodes; all-to-all on routing.
• Activation checkpoint / selective recompute: recompute attention, cache MLP — ~30% compute for 2× memory.
• Overlap FSDP prefetch with compute.
• Fault tolerance: 1M GPUs → MTBF hours. Checkpoint to NVMe every ~30min; async durable upload; redundant pipeline stages.
• Pre-write the loss-spike runbook before launch.
• Liquid-cooling at GW scale → probe thermal failure modes, power-line transients.

1. Health-check daemon + heartbeat; controller declares node dead on miss.
2. Pause all surviving ranks at next iteration boundary.
3. Load latest ZeRO-3-sharded ckpt; dead node's shard held by redundant DP-rank replica.
4. Spin up hot-spare; install recovered shard.
5. Reform process group; resume from ckpt_step + 1.

Trickies: dataloader replay (seed + step), optim momentum is part of ckpt (skipping = post-resume spike), per-rank RNG state, async-ckpt loses last N steps → accept rewind.

At 1M GPUs, expect 30+ recoveries per 30-day run. Hide them from the researcher.

• Parse streamed diff format with byte buffer for mid-line tokens.
• Apply optimistically to shadow buffer; commit at hunk boundaries.
• Conflict: if user typed during streaming, base-hash mismatch → 3-way merge or pause.
• Every accepted hunk = one undo step.
• Backpressure: queue if applying lags network — never block network read.
• User types → abort; drop queued hunks; no partial apply.
• Model regenerates same prefix differently → version your hunks.
• Perceived latency = application latency, not network. Optimistic-render-then-confirm.

• Plan step: LLM produces structured plan (files to read/edit, tests). Schema-constrained.
• Context retrieval: per-file + related symbols via grep/LSP. Token-budget priority: most-recently-modified.
• Tool loop: read_file, write_file, run_command, run_tests. Stage to ephemeral git worktree.
• Verify: tests; failures feed back; cap retries 3-5.
• Failure → revert staging branch; show diff + explanation.
• Dangerous tools (rm, network, secrets) → user confirm.
• Agent state = serializable graph; resumes if Cursor closed.
• Budget tokens × wall-clock; abort early if no progress.
• Failure mode: wrong-file edits → verifier LLM scores plans pre-execution; user approves plan upfront for large changes.

• VPC-deployed inference: ship Docker w/ weights to customer cloud. Pro: code never leaves. Con: ops burden, customer GPU bill, slow updates.
• TEE + attestation: AMD SEV-SNP or Nvidia confidential GPUs; customer KMS releases key only to attested workload. Pro: Cursor maintains. Con: ~10% perf, attestation complexity.
• Differential filtering: minimal context windows (file + imports) + regex-redact secrets pre-send.
• Practical: (3) default + (2) top-tier enterprise.
• Audit: per-tenant logs with model version + request hash; replayable.
• Tab-prediction's <100ms budget makes pure-on-prem costly → most enterprises pick (2) with regional residency.

• Init reservoir with first 3. For each item at 0-indexed i: j = uniform_int(0, i); if j < 3, replace reservoir[j].
• Prob: at step i, any item in reservoir = 3/(i+1) (induction).
• Weighted extension: A-Res / Chao's weighted reservoir.
• Perplexity asks because their retrieval pipeline samples from very large candidate streams under memory caps.

• Registry (provider, client, capabilities); model_class ("fast"/"smart"/"code") maps to candidates.
• Routing: latency p50 + load + user-stickiness (avoid mid-conversation switch).
• Failover: primary → 5s timeout / 5xx → secondary. Circuit-break after N consecutive fails, exponential backoff.
• Per-provider token-bucket rate limit.
• Streaming: failover only before first token; after that, abort or finish — don't switch.
• Idempotency via request_id; retries dedupe.
• Hedging (parallel send, first-response-wins): doubles cost for latency-critical paths.

• Both are E2M1 (1S/2E/1M) with per-block scale. Differences:
• Block size: MXFP4 = 32; NVFP4 = 16. 2× more scales → tighter local dynamic range, ~2× less quant error.
• Scale format: MXFP4 = E8M0 (pow-2 only); NVFP4 = E4M3 (mantissa allows non-pow-2) + FP32 global per-tensor scalar. Two-level scaling.
• Result: <1% degradation on key LM tasks vs FP16; 3.5× memory cut.
• Train in NVFP4: needs Nemotron QAD recipe to recover accuracy.
• Per-block scaling overhead means at batch=1 decode shapes FP8 sometimes wins.

• Maintain (m, d): running max + running sum-of-exps normalized to current max.
• Per new x: m_new = max(m, x); d_new = d × exp(m - m_new) + exp(x - m_new); m,d = m_new,d_new.
• End: s_i = exp(x_i - m) / d.
• FA extension: per K/V block, do online (m,d) update on each Q-row AND rescale running output O ← O × exp(m_old - m_new) because per-block denoms differ.
• CUDA: load Q to SMEM; iterate K,V tiles from HBM; accumulate O, (m,d) in regs/SMEM; write back.
• Preserves IEEE-754 (modulo associativity). FA4 lazy-rescaling skips exp(Δm) when shift < ε.

• Candidate gen: two-tower + PinSage GNN + recent-board candidates. Union ~2000.
• Coarse ranker: small DNN, <30ms, top 200.
• Fine ranker: multi-task DNN predicting [click, save, close-up, hide]. MMoE for task-specific gating over shared experts.
• Calibration: isotonic on holdout.
• Value model: w_click × p_click + w_save × p_save - w_hide × p_hide; weights tuned via counterfactual eval + online A/B.
• Diversity re-rank: DPP / MMR.
• Cold start: pop+content; new pins use visual SigLIP-class embed.
• Offline AUC ≠ online win → replay-buffer sim + interleaving.
• P95 end-to-end ~150ms. Daily training + online learning.

• Fixed: simple, requires tuning, fine when task scales stable.
• Uncertainty weighting (Kendall): learn σ_k; L = Σ (1/2σ²) L_k + log σ. Noisy tasks auto-downweight.
• GradNorm: balance gradient magnitudes across shared backbone.
• PCGrad: project conflicting gradients (neg cosine) to prevent destructive interference.
• Correlated tasks (click+save) → uncertainty weighting. Conflicting (CTR + diversity) → PCGrad. Business priority (revenue ≫ ancillary) → fix.
• Pitfall: aux task with huge numerical loss (dwell-time-in-seconds regression) drowns out small log-loss → rescale by per-task moving avg first.

• Feature store: Redis online + Hudi/Iceberg offline. Point-in-time joins at training to avoid leakage.
• Training: daily incremental + weekly full; Ray Train / Horovod; MLflow with parent-child experiments.
• Model registry: S3 weights + metadata (training-data window, hparams, evals). Canary 5%→25%→100% with auto-rollback.
• Serving: Triton + TensorRT-LLM for deep; CPU for shallow.
• Trust integration: trust classifier in candidate pipeline as hard gate, not re-rank — content failing safety never reaches the user.
• A/B: multi-armed bandit on top; consistent bucketing across experiments.
• Per-bucket metrics with anomaly alarms.

1. INT4 weights + INT8 activations. Per-channel weight scaling, per-tensor (or per-token in attention) activation scaling.
2. Calibration: 256-1024 representative samples; 99.9th-percentile range beats min/max for fat-tail activations.
3. GPTQ or AWQ: AWQ rescales weights by per-output-channel activation magnitude → keeps important channels' bits.
4. KV cache quant: INT8 halves on-device memory; calibrate per layer.
5. QAT/QAD: small recovery LoRA distilled from FP teacher.
6. Core ML compile via coremltools; ANE-supported layer mapping; unsupported → GPU fallback.
7. Profile w/ Instruments → Neural Engine template. Tile sizes matter; batched matmul > many small.
8. Privacy: weights + KV in secure enclave region — no exfil path.
9. Aim <5-20ms/token; 200ms/token unusable on iPhone.

• Detect: ProcessInfo.thermalState.
• Graceful degrade: drop speculative; dense → quantized variant; small-model fallback for completion; cloud offload of final tokens if user opted in.
• Adaptive decoding: greedy (saves draft compute); lower top-k; shorter max length.
• Non-streaming UIs: generate to checkpoint, return partial, queue continuation when temp drops.
• Defer non-critical (autocomplete, summary) to charging-overnight slots.
• Communicate to user — "thinking…" is fine; surprising slowdowns aren't.

• DPO: BT model P(y_w ≻ y_l | x) = σ(r(x,y_w)-r(x,y_l)). KL-constrained RL optimum: π*(y|x) = (1/Z(x)) π_ref(y|x) exp(r/β). Solve for r = β log(π/π_ref) + β log Z. Substitute into BT — Z(x) cancels for pairwise difference. L = -E[log σ(β log π/π_ref|y_w - β log π/π_ref|y_l)]. No reward model needed.
• GRPO: sample G completions per prompt, rewards r_1..r_G, normalize A_i = (r_i-μ)/σ (group-relative advantage, replaces critic). PPO-style clipped objective + KL to reference. No value function.
• DPO: preference pairs only, collect-once; great for chat alignment.
• GRPO: on-policy(-ish), needs reward signal — natural for verifiable reasoning (math/code).
• 2026 trend: GRPO + RLVR displaced DPO for reasoning; DPO still dominant for chat. Wang et al. show GRPO is a DPO special case under reparam.

• Total P, activated P_a = k × P/E. Specialization comes from combinatorial diversity: C(E,k). E=128, k=8 ≈ 1.4e10 modes; E=8, k=2 = 28.
• Each mode = a distinct learned behavior. Routing must find the right mode — empirical at >~256 experts the routing problem hardens.
• Failure modes: (1) expert collapse → mitigate w/ LB loss or V3-style aux-loss-free bias tuning; (2) token-dropping under imbalance — capacity factor; (3) all-to-all comms on every layer — MoE only wins multi-node.
• Shared experts (V3, GLaM): one always-active to handle generic patterns; routed experts specialize on residual.

• QKV proj → reshape (B, n_h, T, head_dim).
• RoPE on Q,K only (not V): inv_freq = 1/(θ^(arange(0,d,2)/d)). Vectorized: x_rot = stack([-x[..., 1::2], x[..., 0::2]], -1).flatten(-2); out = x*cos + x_rot*sin.
• SDPA: att = (q @ k.T)/√d_h + causal_mask; softmax; @ v.
• GQA: n_kv_heads < n_q_heads; repeat_interleave K,V.
• Inference: kv_cache param; concat new rows.
• Training: prefer F.scaled_dot_product_attention (dispatches to FA); interview wants explicit version.

1. Query rewriting (LLM reformulates for retrieval).
2. Hybrid retrieval (BM25 + dense embed-v4) → reciprocal rank fusion. Continuously indexed (news → vector DB).
3. Cross-encoder rerank top-50 → top-5.
4. Citation-grounded answer with explicit "if not in passages, say you don't know."
5. Freshness boost; last-24h preferred for "today" queries.
6. Hallucination check: claim-level NLI against passages.

Failure: retrieval miss → fallback to stale parametric → hallucinate; mitigate w/ explicit refusal on low retrieval confidence. Conflicting sources → source-reliability scoring.

When NOT to use RAG: math, code, anything self-contained — RAG adds noise.

• Retrieval: geo (PostGIS) + cuisine intent → ~500 in <20ms.
• Two-tower coarse offline → top-100 in <30ms via FAISS HNSW. User embed = sessionized RNN, cached in Redis.
• Fine: wide-and-deep, hashed embeddings for new stores. Multi-task heads [order, abandon, dasher-supply].
• Value model = task preds × business weights, including dasher-supply-aware down-weighting when supply tight.
• Diversity re-rank by cuisine + price tier.
• Latency budget: 20+30+60+30+10buffer = 150ms p95.
• Cold start: market = ETA-based fallback; user = geo-popular + cuisine-default.
• Delayed labels (~30min): delayed-reward training via temporal Bayesian attribution.
• Switchback experiments because consumer A/B affects dasher state (SUTVA violation).

• Selection bias: training data conditioned on what current ranker showed. New ranker promotes different items → no labels for them. AUC on logged distribution overestimates agreement, underestimates disagreement.
• IPS: reweight by 1/p_logged; high variance, unbiased.
• Doubly robust: IPS + value estimate; consistent if either correct.
• Replay: score only on agreement samples; bias-free where it works.
• Counterfactual logging: sample randomly from top-K with logged probs for future eval coverage.
• Interleaving: within-user comparison; more powerful than between-user A/B.
• Marketplace: switchback experiments to handle SUTVA.
• Online A/B remains gold standard; offline eval = prioritization.

• E4M3: more mantissa, smaller range — fine for activations after LayerNorm (bounded range).
• E5M2: more exponent, wider range — needed for gradients (vary 10+ orders of magnitude).
• Standard: E4M3 weights/activations, E5M2 gradients.
• Delayed scaling: per-tensor max via EMA; scale computed offline so kernel doesn't take a global max each step.
• Optimizer state (m, v) stays FP32/BF16 — quantizing it is its own research area.
• RMSNorm fine; MoE routing sensitive — tiny noise flips routing → token-drop spikes.

• Naive: p = softmax(logits); loss = -log(p[y]). exp overflows in FP16 (max ≈ 65K, exp(11) > 65K). Subtract max helps, but log(p) near zero loses precision when uncertain.
• Fused identity: log_softmax(x)[y] = x[y] - logsumexp(x) = x[y] - (max + log Σ exp(x-max)). No nested exp+log — just one logsumexp, numerically stable.
• Gradient: p - one_hot(y); no 1/p needed.
• In FP16/BF16: fused is ~4× more accurate at low confidence.