ML system design — 12 worked problems

The signals interviewers grade on

• Did you clarify before architecting? Skipping this is the #1 way to lose Sr Staff loops.
• Did you name trade-offs explicitly? "I'd use HNSW because we're under 100M vectors and want highest recall at the cost of memory."
• Did you cover eval AND monitoring AND gotchas? Junior candidates stop at architecture.
• Did you name second-order effects? Position bias, feedback loops, train/serve skew, calibration drift, network effects in marketplace experiments.
• Did you stay scope-disciplined? When the interviewer asks "deep-dive on the ranker," you stop talking about retrieval.

Recommend videos to 2B users from a corpus of billions, with sub-100ms latency, optimizing for watchtime AND satisfaction (likes, subscribes, no-dislike) — without collapsing into a clickbait filter bubble. The hard part of this problem is the multi-objective optimization: pCTR alone produces clickbait; pWatchtime alone produces engagement-bait; you need a multi-task ranker with the right weights, and you need diversity / exploration baked in.

Requirements

• 2B users, 500h/min uploaded
• Latency < 100ms p99
• Optimize watchtime + satisfaction (likes, subscribes, low dislike rate)
• Avoid filter-bubble + clickbait dynamics

Data

• Implicit: watch, watch_fraction, skip
• Explicit: like, dislike, subscribe, share
• Context: time, device, locale
• Content: channel, topic, language, age
• User: history, social graph

Architecture — classic two-stage funnel

• Candidate generation (retrieval): multiple sources merged.
  • Two-tower: user tower (ID embed + recent history sequence + demographics) and item tower (video features), trained with sampled-softmax / in-batch negatives. ANN (HNSW or ScaNN) over O(100M) videos → top ~1k.
  • Co-watch / collab: classical retrievers.
  • Subscriptions / fresh: from followed channels.
  • Trending / locale popular.
• Ranking: cross-encoder (DLRM-style or transformer). Inputs: user features, candidate features, cross features, sequence of recent interactions (DIN/SASRec attention over user history vs candidate). Multi-task heads: pCTR, pWatchtime, pLike, pComment, pDislike. MMoE for shared+task-specific experts.
• Final scoring: weighted sum across heads (weights tuned via online experiments / Pareto), with diversity / recency / freshness boost. MMR or DPP for diversity.

Training infra

Massive embedding tables (hundreds of GB) — sharded model parallel (TorchRec). Streaming training (Kafka → Flink → trainer) for freshness. Daily full retrain + hourly incremental.

Serving

Retrieval served from in-memory ANN index (replicated). Ranker on GPU/CPU farm with feature store fan-in. Online (Redis-like, sub-ms) + offline (warehouse for training).

Eval

Offline AUC, NDCG, recall@k, calibration. Online: A/B tests with primary metric watchtime, guardrails on dislikes/abandonment/diversity.

Monitoring

Per-head calibration drift, retrieval recall slipping, feature freshness, p99 latency, fraction of recommendations that come from each retrieval source (don't let one collapse), exploration slot fraction.

Like YouTube, but with severe recency, real-time engagement signals, and heterogeneous content (text/image/video/links). The hard part of this problem is real-time freshness: a tweet from 2 minutes ago with 500 likes/min is more valuable than a tweet from 2 days ago with 10k cumulative likes — your ranker needs to consume real-time engagement counters as features without breaking train/serve consistency.

Requirements

• Real-time, sub-200ms p99
• Billions of tweets, heterogeneous signals (text, image, video, links)
• Strong recency requirement — minutes-old content matters

Architecture

• Source mixer: in-network (people you follow), out-of-network (recommended), promoted, communities/lists. Each contributes candidates.
• Heavy ranker: MaskNet / DCN-v2 with multi-task heads (p_like, p_RT, p_reply, p_dwell, p_negative_feedback). Inputs: user-author affinity, real-time engagement counts, embedding similarity, recency decay.
• Heuristic re-rank: diversity (cap per author), recency boost, sensitive content filter.

Training infra

Hours-fresh data; daily retrains. Embeddings for user/author/topic. SimClusters (Twitter's clustering) provides interest representations.

Serving

Real-time engagement counters via Kafka + sliding window aggregator. Per-tweet feature lookup at request time. Hot tweet cache.

Eval

Offline: AUC per head, calibration. Online: dwell time, follow rate, negative feedback rate, retention.

Predict click probability for ad-impression pairs at millions of QPS, with predicted probability used directly in second-price auctions and pacing systems. The hard part of this problem is calibration: an AUC-optimal ranker that's miscalibrated will mis-bid on every impression. CTR is also extremely high cardinality (billions of users × millions of ads × thousands of slots) and very imbalanced (~0.1–1% positive rate).

Requirements

• O(ms) latency at QPS scale of millions
• Very high cardinality features
• Good calibration (predicted CTR feeds auctions and pacing)

Data

Impressions and clicks (positive class very rare, ~0.1–1%), context (page, slot, time), ad creative, advertiser, user features.

Model — classic stack

• Feature embedding tables for IDs (user, ad, publisher). Hash-bucket for OOV.
• DLRM (Naumov 2019, arxiv 1906.00091): bottom MLP on dense, embedding lookup for sparse, pairwise dot product across all sparse pairs → top MLP → sigmoid.
• DCN-v2 (Wang 2020, arxiv 2008.13535): explicit cross terms via low-rank cross network.
• Sequence: DIN/DIEN attention over user's recent ad interactions.

Training infra

Streaming online + daily retrains. Massive embedding tables (TBs) → row-wise parallel.

Serving

Embedding-table lookup service (sharded by hash), feature fetch, ranker eval, calibration, score back to auction.

Eval

AUC, log-loss, normalized cross-entropy (NCE), calibration plots. Online: revenue, eCPM, advertiser ROI.

Calibration — critical for auctions

Apply isotonic regression or Platt scaling on a held-out window; recalibrate frequently. AUC measures ranking, not whether predicted 0.05 actually means 5% click rate. In a second-price auction, miscalibration directly leaks revenue.

Search ranking with personalization, but careful — over-personalizing on ambiguous queries ("Apple") gives the wrong intent. The hard part of this problem is unbiased eval: position bias is enormous in search (top result gets 30%+ of all clicks regardless of relevance), so you need click models or randomized exploration to even measure quality.

Pipeline

• Query understanding: spelling correction, entity linking, intent classification, query embedding (dense retrieval), expansion.
• Retrieval: hybrid — BM25 (lexical, inverted index) + dense (two-tower with query/doc encoders, ANN over corpus). Late-interaction ColBERT for higher recall.
• L1 ranker (cheap, ~1k candidates): linear or shallow GBDT.
• L2 ranker (cross-encoder, ~100): BERT/transformer cross-encoder scoring (query, doc) pair. Listwise loss (LambdaMART / ListNet).
• Personalization: user embedding (long-term + session). Don't over-personalize for ambiguous queries.

Eval

NDCG@k, MRR, click models for unbiased eval (Cascade, PBM, DBN). Online: search success, session abandonment.

Monitoring

Per-segment NDCG (head queries vs tail queries vs navigational vs informational), latency p99, freshness for news queries, click-through patterns by position.

Serve 200M+ DAU at sub-2s TTFT and sub-50ms inter-token latency, across short and long prompts, with multi-tier SLAs and tight GPU economics. The hard part of this problem is head-of-line blocking: a single 100k-token prefill request can block dozens of short requests in a naive scheduler. The solution is disaggregated prefill+decode pools, prefix caching, continuous batching, and SLO-aware scheduling.

Requirements

• 200M+ DAU
• p99 TTFT < 2s, ITL < 50ms
• Mix of short and long prompts
• Multi-tenant SLA tiers (free / plus / pro / API)
• GPU efficiency (cost is the constraint)

Architecture

• Global LB → region → cluster.
• Disaggregated serving: prefill pool + decode pool. KV transferred over RDMA.
• Prefix cache: hash-table keyed on prompt prefix tokens. Block-aligned (PagedAttention).
• Continuous batching: per-iteration scheduling.
• Tensor parallel within node (TP=8 for 70B+); pipeline parallel only for very large models.
• Speculative decoding with EAGLE-style head.
• Quantization: FP8 weights+activations; INT4 weights for cost-sensitive variants.
• Routing: model router selects mini/full/reasoning model based on query complexity.

Eval

Offline (held-out conversations, MMLU/HumanEval/MATH/AgentBench), online (thumbs feedback, conversation length, retry rate, A/B on model versions).

Monitoring

TTFT/ITL distributions, KV cache hit rate, GPU utilization, queue depths, OOM rate, refusal rates, output toxicity.

Build the production pipeline that takes raw user prompts (and red-team prompts) and produces a fine-tuned, aligned, eval-gated, canary-rolled model. The hard part of this problem is the safety-helpfulness trade-off and the regression-gate engineering: a tiny safety regression on one slice should block the rollout, even if helpfulness improves overall.

Pipeline

1. Data ingest: raw prompts (real user, red team, synthetic). Deduped, PII-filtered.
2. Generation pool: base model generates K candidates per prompt; self-critique with constitutional principles; revisions.
3. Preference labeling: AI judge (stronger model) rates pairs against constitution. Sample subset for human review (gold set).
4. Training: SFT on revisions; then DPO/KTO/PPO on preferences. Multiple iterations.
5. Evaluation: capabilities (MMLU, MATH, HumanEval), safety (HarmBench, XSTest, jailbreak suite), instruction following (IFEval), preference winrate vs prior model.
6. Regression gate: blocked from rollout if regression on N safety / capability metrics > ε.
7. Canary rollout: 1% → 10% → 100% with online metrics monitored.

Eval

Capabilities (MMLU, MATH, HumanEval, MMLU-Pro, GPQA), safety (HarmBench attack success, XSTest over-refusal, JailbreakBench), instruction following (IFEval), preference winrate vs prior model (Arena-Hard, AlpacaEval 2).

Monitoring

Online thumbs feedback, refusal rate, output toxicity rate, conversation length, retry rate, jailbreak attempt rate. Per-cohort breakdowns.

Build a retrieval-augmented generation system over billions of documents that produces grounded, cited answers. The hard part of this problem is chunk-boundary loss and "lost in the middle": naive chunking destroys context across boundaries; long stuffed contexts cause LLMs to underuse mid-context information.

Pipeline

• Ingest: chunking (recursive, sentence-aware, ~500–1000 tokens with overlap; or semantic chunking via embedding similarity).
• Embedding: dense (E5, BGE, OpenAI text-embedding-3, Cohere embed v4) with possible Matryoshka representation (truncate dim for cheap recall, full dim for precision).
• Storage: vector DB (Pinecone, Weaviate, Qdrant) or built-in (FAISS sharded, ScaNN).
• Sharding: by tenant, then by embedding cluster (IVF). Replicas for QPS.
• Retrieval: hybrid (BM25 + dense with RRF or learned weighting), large k (~100). Optional ColBERT-style late interaction.
• Reranking: cross-encoder (bge-reranker, cohere-rerank) on top-100 → top-10.
• Augmentation: stitch retrieved chunks into prompt with source markers. Truncate to context budget.
• Generation: LLM with citation requirements (grounded).
• Optional: query rewriting (multiple sub-queries via LLM), chain-of-thought retrieval (multi-hop), HyDE (generate hypothetical answer, embed, retrieve).

Eval

• Retrieval: recall@k, MRR vs labeled relevance.
• End-to-end: faithfulness (RAGAS, TruLens), answer correctness, citation accuracy.
• Hallucination rate: NLI-based check.

Monitoring

Retrieval recall@k drift, citation accuracy, hallucination rate, latency per stage (embed, retrieve, rerank, generate), index staleness per tenant, embedding-model version pinning.

Build a billion-vector ANN system that handles inserts, deletes, attribute filters, and consistent QPS. The hard part of this problem is the filter problem: vectors are easy to find, but combining ANN with attribute filters (tenant, time range, category) breaks the obvious algorithms in non-obvious ways.

Algorithms

• Flat (brute force): exact, O(N·d). Good for <1M.
• IVF: cluster with k-means into C centroids. At query, search nearest probe centroids, scan their lists.
• PQ: split d-dim vector into M subvectors, quantize each with k=256 (1 byte each). Compressed asymmetric distance via lookup tables. IVF-PQ combines both.
• HNSW: hierarchical small-world graph. Multi-layer; greedy descent with ef-search controlling recall. Excellent for in-memory.
• ScaNN (Google): IVF + anisotropic quantization. Strong recall/latency.
• DiskANN (Microsoft): SSD-resident graph; ~2× slower than in-memory HNSW but 10× cheaper.

Memory: billion 768-d float32 = 3 TB; PQ to ~32 bytes/vec → 32 GB.

System

• Sharding: by tenant, then by ID range or by clustering.
• Replicas: for QPS and HA.
• Updates: insertion in HNSW is O(log N); deletion is hard (tombstones + periodic compaction). Re-indexing for embedding model changes.
• Filters: pre-filter (filter then search — bad for low-selectivity), post-filter (search then filter — bad for high-selectivity), inline filter (HNSW with attribute-aware traversal).

Eval

Recall@k against brute-force ground truth on a sample; query latency p50/p99; throughput at target recall.

Monitoring

Per-shard latency, recall sampling against brute-force baseline, index size, deletion-tombstone fraction, compaction lag, filter-selectivity histograms.

Build the feature store every ML team in a company shares: declarative feature definitions, online and offline serving, point-in-time joins for training, governance. The hard part of this problem is train/serve skew prevention and point-in-time correctness: training joins must look up feature values as of the label timestamp, never the current value.

Architecture

• Feature definition: declarative (Feast, Tecton). One source of truth.
• Storage: online low-latency KV (Redis, DynamoDB, Cassandra); offline data warehouse / lake (BigQuery, Snowflake, Iceberg) with full history.
• Compute: batch (Spark/Flink) → write to offline store; reverse-ETL → online. Streaming (Kafka → Flink/Beam) → real-time aggregations. On-demand transforms (compute at request time).
• Point-in-time joins: for training, join labels with feature values as of label timestamp — prevents leakage.
• Serving: feature fetch service fans out lookups, applies on-demand transforms, returns vector to model server.

Governance

Feature ownership, lineage, freshness SLAs, deprecation workflow, cost attribution.

Eval

Per-feature freshness SLA tracking, online/offline parity sampling (compute online, log, recompute offline, diff), feature-vector latency p99.

Build the company's experimentation platform: assignment, logging, metrics, statistical engine, holdouts, heterogeneous treatment effects. The hard part of this problem is SUTVA violations in marketplaces and social networks: two-sided platforms break the "treatment of one user doesn't affect others" assumption, requiring cluster randomization or switchback designs.

Components

• Assignment: deterministic hashing on (user_id, experiment_id) → bucket. Mutually exclusive layers (Overlapping Experiments Infrastructure, Tang 2010 — Google paper). Holdouts.
• Logging: exposure events (when user actually entered the experiment surface), interaction events.
• Metrics: hierarchical — North Star (DAU, revenue), proxy (CTR, sessions), guardrails (latency, error rate).
• Stat engine: Welch's t-test for means; CUPED variance reduction (Deng 2013); sequential testing (mSPRT) for early-stopping safety; Bonferroni / BH for multiple comparisons.
• Heterogeneous treatment effects: causal forests, meta-learners (S/T/X-learners) to find subgroup wins.
• Switchback / interleaving for marketplace effects (network interference).
• Long-term effects: holdouts (a population that never sees treatment for months) to measure novelty decay.

Eval / monitoring

Sample ratio mismatch (SRM) detection; bucket-balance checks; metric stability (rolling baseline drift); experiment-coverage dashboard.

Recommend items where the items are inherently multi-modal: short videos with audio + text + visual + author info. The hard part of this problem is modality fusion + missingness: not every item has every modality, and a model that assumes "always have video" breaks on items without one. Plus, video encoding is expensive — precompute and cache aggressively.

Architecture

• Item encoders: pretrained encoders per modality (CLIP/SigLIP for image-text, video transformer for clips). Project into shared embedding via projection head.
• Multi-modal fusion: gated attention combining modalities; quality of each modality gated.
• User encoder: sequence model over user's historical item embeddings + side info.
• Two-tower retrieval with multimodal item tower; ANN.
• Cross-encoder ranker: full attention between user history and candidate, multi-task heads.
• LLM-augmented (optional): LLM generates natural language query/intent from history; semantic retrieval over multimodal embeddings.

Training infra

Pretrain modality encoders separately; align via contrastive on (user_history, clicked_item); fine-tune end-to-end on engagement.

Serving

Precompute item embeddings on ingest; cache aggressively. Video encoding latency is hostile — never compute on the request path.

Eval

Per-modality ablation: how much does each modality contribute? Cold-start eval on items with subset of modalities. Standard recsys metrics (recall, NDCG, online engagement).

360° real-time perception at 10–30 Hz, sub-50ms latency, multi-sensor (camera + lidar + radar), high recall on safety-critical objects, must generalize to long tail (mattress on highway, construction worker holding a sign). The hard part of this problem is long-tail rare objects + auto-labeling at PB scale — you can't human-label everything, you need an offline-large-model auto-labeler with human verification on edge cases.

Requirements

• 360° perception at 10–30 Hz
• Latency < 50ms
• Multiple sensors (camera, lidar, radar)
• High recall on safety-critical objects
• Generalize to long tail

Architecture

• Sensor fusion:
  • Early: project lidar onto camera, jointly process.
  • Late: per-sensor detections, fuse in BEV.
  • Mid (modern): BEVFormer / Lift-Splat-Shoot — lift camera features into 3D via depth distribution, splat into BEV grid, transformer over BEV.
• Backbone: per-camera ConvNet/ViT; temporal fusion (multiple frames).
• Heads: 3D object detection (bbox, class, velocity), semantic/instance segmentation, lane detection, drivable area, traffic light state, pose, intent prediction.
• Tracking: Kalman / data association; re-ID embeddings.
• Prediction: trajectory prediction (Trajectron, MultiPath, MTR) — multimodal trajectory distributions.
• End-to-end alternatives: Wayve / Tesla FSD V12 — camera → planning, trained on driving demonstration + RL from interventions.

Training infra

PB-scale data; auto-labeling pipeline (large offline model labels, humans verify edge cases); simulation for rare scenarios; mining hard negatives from fleet.

Eval

Offline (mAP, IoU, ADE/FDE); closed-loop sim; shadow mode (parallel run, log disagreements); on-road (disengagement rate).

Monitoring

Per-class recall (especially safety-critical: pedestrian, cyclist), calibration drift between sensors, latency p99, disengagement rate per region/condition, sim-to-real gap.

The phase-by-phase tactic sheet

1. 0–5 min: clarify requirements, scale, what to optimize, what's allowed. Confirm with interviewer. Write the constraints on the board.
2. 5–10 min: high-level architecture (boxes + arrows). State the funnel/stack.
3. 10–25 min: deep-dive 1–2 components the interviewer signals interest in. Discuss alternatives. This is where 50% of your signal lives.
4. 25–35 min: training infra + serving infra + eval (online + offline).
5. 35–45 min: gotchas, monitoring, what would you do differently with more time.

The "if I had more time" close

End every loop with 2–3 things you'd explore further: alternative architectures, deeper eval, harder gotchas. This signals scope-discipline (you knew what you skipped) and intellectual honesty (you don't claim to have solved everything in 45 min).

ML system design quiz — readiness check

1. Walk through YouTube's 2-stage funnel.
   Show answer

   Retrieval (~1B → 1k): two-tower + ANN, plus other sources (collab, fresh, trending). Ranking (~1k → 100): cross-encoder MMoE with multi-task heads (pCTR, pWatchtime, pLike, pDislike). Final scoring + diversity (MMR/DPP) + business rules → top 10.

2. How would you handle cold start for new users / new items?
   Show answer

   New user: content-based recs from session signals (initial interactions, demographics, source); explore via bandits. New item: content-based features (text/image embeddings) drive retrieval; explore via mandatory exploration slots; meta-learning if you have many items per category.

3. Your CTR model has high offline AUC but loses in A/B test. Diagnose.
   Show answer

   Suspect (1) selection bias / exposure bias — train log only contains items the old model showed; (2) train/serve skew — different feature pipeline; (3) miscalibration — AUC measures ranking, not calibrated prob; (4) shift in business mix; (5) novelty: model fits historical patterns that no longer hold.

4. Why does an MMoE outperform a shared-bottom multi-task model?
   Show answer

   Shared-bottom forces all tasks through one trunk → negative transfer when tasks conflict. MMoE has multiple experts shared across tasks; per-task gates softly mix experts. Each task can pick its preferred expert combination → less interference.

5. How do you debias position effects in a learned ranker?
   Show answer

   (1) PAL / position-aware learning: feed position as a feature during training; set to "no position" at inference. (2) Two-tower: shallow tower predicts position effect; main tower predicts relevance. (3) Click models (PBM, Cascade, DBN). (4) Result randomization for unbiased data.

6. In-batch negatives vs explicit hard negatives — tradeoff?
   Show answer

   In-batch: cheap, popularity-biased (high-frequency items appear more as negatives). Hard negatives (top non-positive from ANN): faster learning per example, risk of false negatives (true positives that look similar). Mixed Negative Sampling (MNS): combine in-batch + uniform-random; logQ correction for popularity debiasing.

7. HNSW vs IVF-PQ — when each?
   Show answer

   HNSW: best recall/latency in memory; high memory (no compression). Use for < 100M vectors with quality SLA. IVF-PQ: compressed (32×–128×); scales to billions; recall lower. Use for billion+ vectors at modest QPS. ScaNN/DiskANN intermediate.

8. Your retrieval layer returns the same 1000 items for everyone. What's wrong?
   Show answer

   Likely: (1) user tower collapsed (output ≈ constant — check embedding norm and variance); (2) lack of personalization features in user tower; (3) popularity-biased sampled-softmax without logQ correction → all items pushed toward popular ones; (4) ANN index returning popular items only (debiasing lost in serving).

9. How do you calibrate a multi-task ranker where each task has different positive rates?
   Show answer

   Per-task isotonic regression on a held-out window. For each task head's logit, fit monotonic mapping logit → calibrated prob using actual positive rate per quantile. Recalibrate frequently (weekly) under distribution shift. Joint calibration if tasks are correlated.

10. Migrate DLRM-style ranker to generative recommender (TIGER/HSTU) — risks?
    Show answer

    (1) Loss of explicit business signal control (multi-task heads gone). (2) Auto-regressive serving latency. (3) Cold-start changes (semantic IDs from RQ-VAE replace ID embeddings). (4) Training-time interactions with online behavior (counterfactual eval harder). (5) Capacity for personalization in generative model still being scaled out. Run as A/B with strict guardrails.

11. Design a RAG system at billion-doc scale.
    Show answer

    Chunking (semantic, ~500-1000 tok with overlap) → embedding (SigLIP/E5/Cohere) → vector DB (sharded by tenant, IVF-PQ within shard) → hybrid retrieval (BM25 + dense, RRF) → cross-encoder reranker → LLM generation with citations. Eval: recall@k, RAGAS faithfulness, citation accuracy.

12. Design a feature store with point-in-time correctness.
    Show answer

    Online (sub-ms KV) + offline (warehouse with full history). Streaming compute (Flink/Beam) → online + offline. For training, point-in-time joins: for each (entity, label timestamp), look up feature value as-of the label time. Prevents leakage of future info into training features.

13. What's CUPED?
    Show answer

    Variance reduction for A/B tests (Deng 2013). Adjust the outcome by a pre-experiment covariate: Y' = Y − θ (X − E[X]) where X is correlated with Y. Reduces variance → smaller sample size for same power. Used at every major web company.

14. How do you avoid SUTVA violations in marketplace experiments?
    Show answer

    SUTVA = stable unit treatment value assumption — treatment of one unit doesn't affect others. Violated by network effects (social) or supply-side competition (DoorDash, Uber). Solutions: cluster randomization (region-level), switchback (alternate treatment over time), counterfactual interleaving for ranking.

15. Design ChatGPT serving for 200M DAU.
    Show answer

    Disaggregated prefill + decode pools. Prefix caching (system prompts shared). Continuous batching. TP=8 within node for 70B+. Speculative decoding. FP8 weights. Model routing (cheap vs reasoning). Multi-tier autoscale. Safety filtering pipeline pre/post inference. SLO-aware scheduling.