ML system design — feeds, search, ads, RAG, and every infra layer underneath

Back-of-envelope: home-feed read path

• DAU ≈ 100M. Suppose each opens the app ~5 sessions/day, each session triggers ~3 feed fetches → 1.5B feed requests/day.
• Avg QPS = 1.5e9 / 86,400 ≈ ~17k QPS. Peak is typically 2–4× average → ~50–70k QPS design target.
• Each feed fetch returns ~25 posts. If we rank ~500 candidates per request, that's ~8.5M ranker scorings/sec at peak — this is why ranking must be batched on GPU and retrieval must be cheap ANN.
• Latency budget: users feel jank above ~150–200ms for a scroll. Target p99 ≈ 150ms end-to-end for the feed.

Write path: votes & posts

• Posts: ~1M/day ≈ 12/sec avg, ~50/sec peak — trivial for the datastore; the cost is downstream fan-out and indexing.
• Votes: the heavy write. If each DAU casts ~20 votes/day → 2B votes/day ≈ 23k QPS avg, ~70k peak. Votes are the canonical case for an async queue (RabbitMQ) — you ack the user instantly and apply the vote behind the scenes.
• Comments: say 50M/day ≈ 600/sec avg. Each comment is a small row but participates in nested-tree reads.
• Engagement events (impressions, dwell, scroll, click, hide): the firehose. Easily tens of millions/sec at peak across all surfaces — this is exactly the Kafka workload Reddit confirms (500+ brokers, tens of millions of msgs/sec).

Storage napkin

• Posts: 1M/day × 365 × ~3 KB ≈ ~1 TB/year of post rows (text + metadata; media in object storage/CDN separately).
• Engagement log: tens of millions/sec × ~200 bytes is petabyte-scale/year — this lives in a data lake / columnar store, with Kafka retention measured in days, not forever.

Windowing

You aggregate over time slices: tumbling (fixed, non-overlapping, e.g. votes-per-minute), sliding/hopping (overlapping, e.g. clicks in the last 5 min, updated every 1 min), session (group by gaps of inactivity, e.g. one browsing session). For a feed, "engagement velocity in the last 10 minutes" is a sliding window.

Event time vs processing time & watermarks

Events arrive late and out of order (mobile networks, retries). Event time = when it happened; processing time = when we saw it. A watermark is the processor's assertion "I believe I've now seen all events up to time T" — it lets a window fire (emit results) while tolerating bounded lateness. This is the whole reason Flink exists: correct windowed aggregates under out-of-order arrival.

Stateful operators & checkpoints

A per-user counter is state. Flink keeps it locally (RocksDB-backed) keyed by the partition key and checkpoints it to durable storage so a crashed task restores exactly. Reddit confirmed: Flink Stateful Functions run Lua rules for content-safety/moderation — small stateful actors evaluating rules over the stream in real time.

Partition count for the engagement firehose

Suppose peak 20M events/sec, each ~200 bytes → ~4 GB/sec ingress. A single partition sustains roughly tens of MB/sec write comfortably (call it ~30 MB/s to leave headroom). So you need on the order of 4 GB/s ÷ 30 MB/s ≈ ~130 partitions just for throughput, and you'd round up (e.g. 256) for consumer parallelism and future headroom. With RF=3 the on-broker write amplification is 3×, so ~12 GB/s of replicated write spread across 500+ brokers ≈ tens of MB/s/broker — comfortable. This is exactly why Reddit runs 500+ brokers.

Retention & disk

4 GB/s × 86,400 s/day ≈ ~350 TB/day of raw events; at RF=3 that's ~1 PB/day on disk. So retention is measured in days, not weeks — you keep ~3–7 days for replay/backfill and sink the long tail to S3/parquet (cheap, columnar) for training. Confirm the consequence: "Kafka is the replay buffer, the lake is the archive."

Consumer count

If a single Flink subtask processes ~100k events/sec, 20M/sec needs ~200 parallel subtasks — which is why you provisioned ≥200 partitions (parallelism is capped by partitions). The arithmetic ties partition count, broker count, and consumer parallelism into one consistent story.

Thundering herd / cache stampede

A hot key expires (or the cache flushes) and thousands of concurrent requests all miss simultaneously and hammer the DB with the same expensive recompute → DB falls over. Fixes, in order of preference:

• Request coalescing / single-flight: only the first miss recomputes; everyone else waits on that one in-flight result. (Also called "promise caching" / "dogpile lock".)
• Early/probabilistic recompute: refresh the value before it expires, with a probability that grows as TTL approaches — so it's renewed by one request while the old value still serves the rest.
• TTL jitter: add randomness to TTLs so 10,000 keys don't expire at the same instant.
• Stale-while-revalidate: serve the stale value and recompute in the background.

Hot key

One key (a viral post, r/popular front page) gets so much traffic that the single shard/node holding it is saturated. Fixes: replicate the hot key across nodes, add a small local in-process cache in front of the shared cache (L1/L2), or shard the value (split a counter into N sub-counters summed on read).

Cache penetration & avalanche

Penetration: requests for keys that don't exist bypass the cache and always hit the DB — fix with negative caching (cache the "not found"). Avalanche: mass simultaneous expiry or a cold cache after restart — fix with jitter and warm-up.

Sizing the feed cache

Cache the ranked feed (just ~25 post IDs + scores ≈ 1 KB) for active users with a short TTL. If ~20M users are active in a TTL window and each entry is ~1 KB → ~20 GB of cache — a handful of Redis nodes. Compare to caching hydrated feeds (post bodies, ~50 KB each) → ~1 TB; so you cache IDs, not payloads, and hydrate fragments from a separate object cache. That decision falls straight out of the arithmetic.

Hit rate and the latency it buys

If cache p50 read = 1ms and DB/compute p50 = 40ms, effective latency = h·1 + (1−h)·40. At 95% hit rate that's ~3ms; at 80% it's ~9ms; at 50% it's ~20ms. So a 15-point hit-rate drop triples latency. Consequence: protect hit rate aggressively (stampede control, warm-up after deploys, jittered TTLs) — and quote effective latency as a function of hit rate, not a single number.

Working-set rule of thumb

Aim to fit the hot working set (Zipf-distributed: a small fraction of posts gets most reads) in RAM. For Reddit, the front-page / r/popular hot set is tiny relative to the catalog, so a modest cache captures most reads — quantify "the top ~1% of posts serve ~50%+ of impressions" to justify the cache size.

The bug

The feed returns 25 posts. Each Post.author resolver naively does SELECT * FROM users WHERE id = ?. That's 1 query for the feed + 25 for authors = N+1 queries. Add subreddit and it's 1 + 25 + 25. This is the classic GraphQL footgun.

The fix: DataLoader (per-request batching + caching)

A DataLoader collects all the IDs requested within a single tick of the event loop and issues one batched query (SELECT * FROM users WHERE id IN (...)), then distributes results back to each caller. It also de-dupes within the request (two posts by the same author → one fetch). Net: N+1 collapses to a small constant number of batched calls.

Data model: query-first, denormalized

Unlike Postgres, you do not model entities and join later. You model queries: design the table so the read you need is a single partition lookup. The partition key determines which node holds the data; the clustering key orders rows within a partition. Denormalization (storing the same data multiple ways) is expected — disk is cheap, scattered reads are not.

-- "give me a user's feed entries, newest first"
CREATE TABLE feed_by_user (
  user_id      uuid,
  bucket       int,          -- time bucket to cap partition size
  created_at   timestamp,
  post_id      uuid,
  score        double,
  PRIMARY KEY ((user_id, bucket), created_at)
) WITH CLUSTERING ORDER BY (created_at DESC);

Tunable consistency

Per-query you choose consistency level: ONE, QUORUM, ALL. The classic rule: if R + W > N (read replicas + write replicas exceed replication factor) you get strong consistency for that operation; otherwise eventual. So QUORUM reads + QUORUM writes at RF=3 (2+2>3) gives strong consistency while tolerating one node down. This is CAP made tunable.

When to pick Cassandra over Postgres

• Write throughput beyond a single primary (votes/engagement at huge scale).
• Predictable single-partition access patterns (no ad-hoc joins).
• Multi-region, always-writable (no single point of failure).
• Avoid when you need ad-hoc queries, transactions across partitions, or strong relational integrity — that's Postgres.

Why ANN

Retrieval scores a query embedding against millions of item embeddings. Exact nearest-neighbor is O(N·d) — too slow. Approximate nearest neighbor (ANN) trades a little recall for orders-of-magnitude speed. Reddit evaluated Milvus, FAISS, Vertex AI Vector Search, and Solr's ANN industry/eval.

HNSW vs IVF-PQ

Tuning knobs: HNSW's efSearch/M trade recall vs speed; IVF's nprobe (cells searched) trades recall vs speed. Always quote a recall@K vs p99 latency operating point — that's the senior detail.

The problem CDC solves

You commit a post to Postgres and also need it in search, the cache, and a Cassandra read-model. The naive dual-write (app writes DB then writes the others) is broken: if the second write fails after the first commits, the systems diverge with no clean rollback. Change Data Capture fixes this by making the DB's commit the single source of truth: a connector tails the write-ahead log (Postgres logical replication slot) and emits each committed row change as an event to Kafka. Downstreams (search index, cache invalidator, derived stores) consume that ordered stream. Reddit reportedly uses Debezium single source — hedge.

Why it's better

• Exactly the committed changes, in order — no phantom writes, no lost updates; the log is the truth.
• Decoupled & replayable — add a new consumer (a new search index) later and replay history; no app change.
• This is the "outbox/CDC" pattern that turns "keep N systems in sync" into "everyone is a Kafka consumer of one ordered change stream."

Catch: schema changes must be handled in the connector; and ordering/at-least-once means consumers still need to be idempotent (ch2).

Postgres write ceiling → why votes are async

A single Postgres primary handles on the order of a few thousand write transactions/sec before lock/WAL contention bites. Peak votes ≈ 70k/sec (ch1) → 10–20× over a single primary. Consequence (say it): "votes can't be synchronous row-updates on the post score — enqueue to RabbitMQ, aggregate in Redis/Cassandra, persist periodically; Postgres only sees low-rate aggregated writes." The number forces the architecture.

Comment storage growth

50M comments/day × ~1 KB × 365 ≈ ~18 TB/year of comment rows — fine for partitioned Postgres or Cassandra, but the read pattern (subtree fetch) drives the storage choice (materialized path for range-scan), not the volume.

Replica count for read QPS

If a hot read is 17k QPS (feed) and one replica serves ~5k QPS comfortably, you need ~4 replicas plus the cache absorbing 90%+ — so really ~1–2 replicas behind a 90%-hit cache. Show how the cache (ch3) collapses the replica count; that's the integrated answer.

A feature store has two faces backed by one definition:

• Online store (Redis/KV/Cassandra): sub-10ms point reads of the latest feature values for the ranker at request time.
• Offline store (warehouse/lake): historical feature values for point-in-time-correct training joins (no label leakage).

The whole reason it exists: compute a feature once from the same definition so the value at training time equals the value at serving time. The #1 silent killer of RecSys models is train/serve skew — a feature computed differently online vs offline. Naming this is a strong senior signal (more in ch13).

How many ranker GPUs?

Peak ~50–70k feed QPS, each request ranks ~500 candidates → ~25–35M item-scorings/sec at peak. Suppose a ranker GPU does ~1M scorings/sec at the batch size that fits the latency budget → you need ~25–35 GPUs of headroom just for ranking, then ×~2 for redundancy/zone spread/burst → ~50–70 GPUs. The exact number is unimportant; showing you can derive it from QPS × candidates ÷ per-GPU throughput is the signal.

Dynamic batching math

At 50k QPS into a fleet, each GPU sees ~700–1000 req/sec. A max_queue_delay of 3ms naturally accumulates a batch of ~3 requests' worth of candidates — enough to fill the GPU without adding more than 3ms of queueing. Push the delay too high and you blow p99; too low and GPU utilization (and cost) tanks. State the dial: "I tune max_queue_delay against the p99 budget to trade a few ms of latency for multiples of throughput."

Embedding table memory

100M users + 1M posts/day retained × a 128-dim float32 embedding ≈ 100M × 512 bytes = ~50 GB just for user embeddings → doesn't fit one GPU's RAM, so embeddings live in a sharded parameter server / KV (CPU RAM), looked up and fed to the GPU ranker. This is why embedding lookup is often the memory bottleneck, not the dense net.

Embedding-table memory, derived twice

User table: 100M users × 128-dim float32 = 100M × 512 B ≈ ~50 GB. Item table (say 90 days of posts ≈ 90M items) × 256-dim = 90M × 1 KB ≈ ~90 GB. Neither fits one GPU (24–80 GB), so embeddings live in a sharded parameter server / KV in CPU RAM, looked up per request and fed to the dense GPU net. Consequence: the embedding lookup + network transfer, not the dense FLOPs, is usually the serving bottleneck — so quantize embeddings (fp16/int8), cache hot rows, and co-locate the PS with the ranker.

ANN index sizing

90M item vectors × 256-dim fp32 = ~90 GB raw. HNSW adds a graph (~1.5–2× memory) → ~150–180 GB → shard across nodes or compress with IVF-PQ (e.g. PQ to 64 bytes/vector → ~6 GB, recall trade). Say the dial: "I'd start HNSW for recall, move to IVF-PQ if memory cost dominates, and quote a recall@100 vs p99 operating point rather than a single latency."

Feature freshness vs cost

Streaming a feature (always-on Flink) costs ~10–50× a nightly batch recompute of the same feature. So tier by need: "comments in last 5 min" must be streamed (seconds-fresh); "30-day topic affinity" is batch (hours-stale is fine). Only stream what genuinely needs second-level freshness — that single decision is a real cost lever a Staff engineer owns.

Step 1 — Clarify & objective

"The home feed is the personalized, ranked list a logged-in user sees — a blend of their subscribed communities plus recommended communities/posts. We optimize for healthy engagement: a multi-task blend of upvote / comment / dwell / share, with explicit negatives (hide / downvote / report / 'not interested'), and we defend long-term retention & community health as the guardrail. We surface ~25 posts per fetch and must feel instant on scroll."

Step 2 — Requirements & scale math

Functional: return ranked top-K personalized posts; paginate infinitely; update with new content and on user actions; gate unsafe content. Non-functional: p99 ≈ 150ms, high availability (feed must never be empty — degrade gracefully), freshness within seconds–minutes, cost-bounded.

Scale (from ch1): ~100M DAU → ~1.5B feed requests/day → ~17k QPS avg, ~50–70k peak. ~500 candidates ranked/request → millions of scorings/sec → ranking must be GPU-batched, retrieval cheap ANN. Vote firehose ~70k peak → async. Engagement firehose tens of millions/sec → Kafka.

Step 3 — ML framing (the funnel)

Label: implicit feedback — positives (upvote, comment, long dwell, share), negatives (hide, downvote, report, fast skip). Training: batch-train the ranker daily/continuously on logged interactions joined point-in-time with features. Hot-score baseline as a strong non-ML floor and a feature.

# Reddit "Hot" ranking (CONFIRMED) — a baseline AND a feature
score(ups, downs, t_post) =
    sign(ups - downs) * log10( max( |ups - downs|, 1 ) )
  + (seconds_since_epoch(t_post)) / 45000
# 45000s ≈ 12.5h half-life: a 24h-old post needs ~10x the net votes
# to match a fresh one. Top = score with NO time decay.
# Rising = upvote velocity. Controversial = high vol, near-equal up/down.

Step 4 — Data & feature pipeline

"Clients emit impressions/clicks/dwell/votes/hides to a Kafka topic partitioned by user_id. A Flink job keeps sliding-window aggregates (engagement velocity, recent topic affinity, session signals) and writes them to the online feature store (sub-10ms). The same topic sinks to the lake for batch features and training joins. Item-side features (post age, sub, content embeddings, early-velocity) and counters come from the same store." This is ch2 + ch6 applied.

Reddit-confirmed signals to name explicitly: account-age trust, the user's tendency to like new communities, their upvote/comment history, and engagement velocity on the post. Reddit publicly emphasizes velocity + account-age trust for the home feed.

Step 5 — Model

Candidate generation (multi-source union)

• Subscribed: hot/top posts from the user's subscribed subreddits (sorted sets in Redis by hot-score).
• Recommended communities: posts from subs the user is likely to join (graph + affinity).
• Embedding ANN: nearest posts to the user's interest embedding (HNSW/IVF-PQ). two-tower is standard industry retrieval; at Reddit it's confirmed for notifications, not asserted for the home feed — present it as "standard two-tower-style embedding retrieval as one source."
• Freshness / exploration: a slice of new posts to avoid feedback-loop ossification and to gather labels.

Ranker (multi-task DNN)

A shared-bottom or MMoE network over user × item × context features with multiple heads: P(upvote), P(comment), P(dwell>t), P(share), P(hide). Combine into a single score with tuned weights: score = w1·P(up) + w2·P(comment) + w3·P(dwell) − w4·P(hide) + …. Multi-task sharing improves data efficiency and lets you trade objectives in re-rank. Feed the hot-score and velocity as features.

Re-ranker

Diversity (MMR / determinantal — don't show 8 posts from one sub), freshness boost, integrity gate (drop/demote unsafe — ch12), frequency caps (don't re-show seen posts), business rules.

Step 6 — Serving architecture & APIs

 client ──▶ Fastly CDN / edge ──▶ GraphQL router ──▶ homeFeed resolver
                                                  │
                                                  ▼
                                       Feed service
                                        1) check per-user feed cache (Redis, short TTL) ── hit ─▶ return
                                        2) miss:  candidate gen (Redis sorted sets + ANN)
                                        3) hydrate features from online feature store (sub-10ms)
                                        4) ranker on model server (dynamic batching, GPU)
                                        5) re-rank + integrity gate
                                        6) write feed cache, return post IDs
                                                  │
                  hydrate post/author/sub/counts via DataLoaders across Go subgraphs
                  (counts/fragments from memcached; truth in Postgres)
    

"The feed is exposed as a GraphQL homeFeed field. Its resolver calls the ranking service for ordered IDs, then hydrates posts through DataLoaders across our federated Go subgraphs — batched to avoid N+1. Counts and rendered fragments come from memcached; source of truth is Postgres."

Step 7 — Scale / latency / caching / reliability (over-invest here)

• Feed cache: precompute & cache the ranked feed per active user with short TTL (tens of seconds) → most reads are O(1); compute-on-read on miss. (ch3: hybrid, not pure fanout.)
• Stampede protection: request coalescing on cache miss for hot users; TTL jitter; stale-while-revalidate so the feed is never empty.
• Hot keys: r/popular & viral posts get replicated cache + local L1; counters split + summed.
• Graceful degradation: if the ranker times out, fall back to the hot-score ordering of cached candidates — degraded but never blank. Kill switch to disable the ML ranker entirely.
• Backpressure: Kafka absorbs engagement spikes; if the feature job lags, serve last-known features (bounded staleness), don't block the read path.
• Latency budget: 5ms edge + 10ms context + 15ms candidate gen/ANN + 12ms feature hydration + 30ms ranker + 5ms re-rank ≈ within 150ms p99.

Step 8 — Eval, A/B, monitoring, feedback

• Offline: per-head AUC/PR, calibration, NDCG/recall@K for retrieval.
• Online A/B: primary = session length / DAU / dwell; guardrails = downvote rate, hide/report rate, complaints, long-term retention. Never ship if guardrails regress even with engagement up.
• Monitoring: p99 per stage, cache hit rate, candidate-gen recall, feature freshness lag, train/serve skew checks, prediction-distribution drift.
• Feedback loop: today's impressions + outcomes become tomorrow's labels; watch for feedback-loop bias (we only learn about what we showed) → keep an exploration slice + log propensities for unbiased eval.

Cold start & integrity

New user: no history → rely on onboarding topic picks, popular-by-locale/age, account-age-trust priors, and exploration; quickly personalize as signals arrive. New post: no engagement → lean on author/sub priors, content embeddings, and a freshness/exploration slice to gather early signal (Rising surfaces velocity). Integrity: every candidate passes a safety gate before ranking output (ch12) — unsafe content is demoted/removed, not just down-weighted.

Step 1 — Clarify & objective

"Detect and act on policy-violating content — spam, toxicity/harassment, NSFW-in-wrong-place, ban evasion, and severe categories (CSAM, violent threats) — across posts, comments, votes, DMs, and reports, in near real time. The true objective isn't 'max recall': it's protect the community while minimizing wrongful removals ('Remember the Human'). So I optimize per-category for an explicit precision/recall operating point and treat wrongful-removal rate and time-to-action on severe content as guardrails."

Step 2 — Requirements & scale math

Functional: score every content event; take graded action (allow / demote / quarantine / remove / route-to-human); support fast rule updates for emerging attacks; feed human decisions back into training. Non-functional: inline decision latency low enough not to delay posting (target ~tens of ms for the rule layer), high recall on severe categories, auditable decisions.

Scale: posts ~1M/day, comments ~50M/day, votes/DMs/reports add more — call it order 10^8 content events/day ≈ ~1–2k/sec avg, ~5k+ peak for content needing classification (far below the engagement firehose, but each is higher-stakes). Severe-category detectors run on 100% of media uploads.

Step 3 — ML framing

A cascade, cheapest-first: (1) deterministic Lua rules + velocity/burst heuristics (catch obvious spam at near-zero cost); (2) a fast ML classifier (toxicity/spam/NSFW) on what survives; (3) heavyweight detectors (hash-match + classifier for CSAM/known-bad) where recall must dominate; (4) a confidence band routes the uncertain middle to humans. Labels: human-moderator/admin decisions, user reports (noisy), and rule hits — heavily class-imbalanced, so PR-AUC not accuracy. Frame each tier as its own precision/recall problem.

Step 4 — Data & feature pipeline

"Content events publish to Kafka. Flink Stateful Functions consume them, keeping per-account/per-IP/per-fingerprint state (posting velocity, recent-removal count, account-age, karma) so behavioral features are computed in-stream. The Lua rules and the ML classifier both read this state." Behavioral features (velocity, account age, network/graph co-activity) catch spam that content models miss — spam is often a behavior, not a string. Human decisions sink back to the training lake.

Step 5 — Model

• Text classifier: a transformer (distilled for latency) over content + a behavioral feature head (account age, velocity, report counts, graph signals). Multi-label across policy categories.
• Image/media: NSFW classifier + perceptual-hash match against known-bad sets (severe categories); OCR to catch text-in-image evasion.
• Coordination detection: graph clustering over accounts/IPs/devices to catch brigading and bot rings (a model over the interaction graph, run in batch + streaming).
• Lua rules layer: human-authored, hot-swappable rules for known patterns — the fast-response weapon between model retrains.

Step 6 — Serving architecture & APIs

 content event ─▶ Kafka ─▶ Flink Stateful Functions ─────────────────▶ decision tier
  (post/comment/        │         Lua rules (hot-swap) + velocity/state      │
   vote/DM/media)       │                     │ survives rules                  ├─▶ allow   (high-conf safe)
                        │                     ▼                                 ├─▶ demote/quarantine
                        │            ML classifier (toxicity/spam/NSFW)       ├─▶ remove  (high-conf bad, high precision)
                        │             + media hash-match                        └─▶ human review queue (uncertain band)
                        │                                                                │
                        └──────────────────────── feedback (labels) ◀────────── moderators/admins
    

The classifier is served behind the stream as a low-latency model service; the review queue is prioritized by severity × confidence so humans see the highest-impact uncertain items first. Every action is logged with the deciding rule/model version for appeals and audit.

Step 7 — Scale / reliability / adversarial

• Adversarial robustness: attackers obfuscate (l33t-speak, homoglyphs, text-in-image), spin up throwaways, and brigade. Defenses: account-age/reputation gating, rate limiting (token bucket — a confirmed Reddit interview topic), OCR + normalization, graph coordination detection, frequent retrains, and hot-swappable Lua rules to respond in minutes without a model redeploy.
• Fail-safe direction: if the classifier is down, the rules layer + conservative defaults still run; for severe categories, fail toward review (queue it) rather than fail-open. Kill switch per detector.
• Cost: the cascade keeps the expensive transformer off the 99% obvious cases; severe-category detectors are the costly always-on tier.

Step 8 — Eval, A/B, monitoring, feedback

• Offline: precision/recall & PR-AUC per category & per action tier; wrongful-removal rate on a human-audited sample.
• Online: shadow new models on live traffic before they take action; A/B on prevalence-of-bad-content-seen, appeal-overturn rate, moderator workload, time-to-action on severe content.
• Monitoring: drift (attackers shift the distribution constantly), queue depth/latency, rule-hit rates, false-positive complaints.
• Feedback loop: human decisions are the gold labels → continuous retrain; but watch label feedback bias (you only get labels on what you surfaced/removed) and audit a random hold-out to estimate true prevalence.

Step 1 — Clarify & objective

"Pick which ads to show a user, where, and at what price, to maximize long-term platform value = advertiser ROI (so they keep spending) + user experience (so ads don't drive users away) + revenue. The proxy I optimize is expected value of an impression = bid × P(action), but I defend user-experience guardrails (ad hide/negative-feedback rate, session length) so we don't over-monetize into churn."

Step 2 — Requirements & scale math

Functional: for each ad slot, retrieve eligible ads (targeting + budget + policy), predict pCTR/pCVR, run an auction, apply pacing, blend with organic. Non-functional: same feed p99 (~150ms, ads add a small slice), budgets never overspend, predicted probabilities calibrated. Scale: feed QPS ~50–70k peak; each request has a few ad slots; candidate ads per request after targeting filters is modest (hundreds–thousands), so the ranker load is smaller than organic but the auction + pacing logic is the new complexity.

Step 3 — ML framing

Two (or more) predicted probabilities per (user, ad): pCTR = P(click) and, for performance ads, pCVR = P(conversion | click). eCPM-style rank: for a CPC ad, rank score = bid × pCTR; for CPA/oCPM, ≈ bid × pCTR × pCVR — the expected revenue per impression. Labels: clicks (dense, fast) and conversions (sparse, delayed — needs delayed-feedback handling). These models must be calibrated, because the predicted probability directly sets price and billing, unlike organic ranking where only the order matters.

Step 4 — Data & feature pipeline

Impression/click/conversion events → Kafka → features. Conversion attribution is the hard part: conversions arrive hours–days after the click (delayed feedback), so the training pipeline must wait/window and correct for not-yet-arrived positives (e.g. delayed-feedback loss). Features: user interest + context (organic), ad/creative features, advertiser/campaign features, and crucially cross features (user × ad category). Budget/pacing state lives in a fast store (Redis) updated in near-real-time off the spend stream.

Step 5 — Model

• pCTR/pCVR model: a DNN with feature crosses (DCN / DeepFM-style) or wide-and-deep; embeddings for user/ad/advertiser IDs. Often multi-task (CTR + CVR with a shared bottom; ESMM-style to handle the click→conversion funnel and CVR sample-selection bias).
• Calibration layer: isotonic regression / Platt scaling on top so predicted probabilities match observed rates — load-bearing because price = f(predicted prob).
• Auction: rank by expected value; charge generalized second price (winner pays just enough to beat the runner-up) — truthful-ish bidding, advertisers don't overpay.

Step 6 — Serving architecture & APIs

 ad request ─▶ targeting/eligibility ─▶ retrieve candidate ads ─▶ pCTR/pCVR model ─▶ auction ─▶ pacing/budget ─▶ blend w/ organic
                (audience, geo,            (Redis/ANN over          (calibrated)        (GSP:           (don't over-       (interleave; cap
                 budget>0, policy)          ad embeddings)                              2nd price)       spend; smooth)      ads density)
    

The blended response goes through the same feed/GraphQL path; ads are interleaved at fixed/learned positions with a frequency cap. Spend is decremented from the budget store on each charged event.

Step 7 — Scale / pacing / reliability

• Budget pacing: spend a campaign's daily budget evenly over the day (probabilistic throttling / PID controller on spend rate), not all in the first hour — protects advertiser ROI and inventory. Pacing state must be near-real-time consistent enough to avoid overspend; small overspend tolerated, large overspend is a financial bug.
• Calibration drift: monitor predicted-vs-actual CTR continuously; recalibrate on drift, because miscalibration directly mis-prices.
• Cold start: new ad/campaign has no history → exploration budget + prior from similar creatives/advertisers; explore-exploit (e.g. Thompson sampling) to learn pCTR without burning advertiser money.
• Reliability: never serve a policy-violating or over-budget ad; fail to organic-only if the ads stack is down (the feed still renders).

Step 8 — Eval, A/B, monitoring

• Offline: CTR/CVR AUC and calibration (log-loss, reliability curve, calibration error) — calibration is non-negotiable for ads.
• Online A/B: revenue / RPM and advertiser ROI and user guardrails (ad-hide rate, session length, long-term retention). Optimizing revenue alone burns the user base.
• Monitoring: calibration drift, budget pacing accuracy, auction price distributions, conversion-delay distribution.

Different advertisers buy different events (clicks, impressions, conversions) and bid different amounts — you can't compare $5-per-click to $8-per-1000-impressions directly. eCPM converts every bid into one currency: expected revenue per 1000 impressions.

• CPC ad: $\text{eCPM} = \text{bid}_{\text{cpc}} \times p\text{CTR} \times 1000$ — you only earn the bid when a click happens, so multiply by click probability.
• CPM ad: $\text{eCPM} = \text{bid}_{\text{cpm}}$ — they pay per impression regardless, so it's already in eCPM units.
• oCPM / CPA (conversion): $\text{eCPM} = \text{bid}_{\text{cpa}} \times p\text{CTR} \times p\text{CVR} \times 1000$ — earn the bid only if click and conversion happen.

If the winner paid their own bid (first-price), everyone would constantly shade bids down and re-bid — unstable, and you have to guess the right shade. The Vickrey / second-price insight: charge the winner just enough to beat the runner-up. Then bidding your true value is optimal (a dominant strategy) — you never pay more than you had to, so there's no reason to game it.

For one slot, the winner is the highest-eCPM ad; the price per click is set so the winner's charged eCPM equals the runner-up's eCPM:

Multi-slot GSP: with $K$ slots, sort by eCPM, slot $i$ goes to bidder $i$, and bidder $i$ pays the price that would just keep them above bidder $i{+}1$. (GSP is not truthful with multiple slots — that's the theoretical wart VCG fixes.)

Two more terms turn the toy formula into a real one:

• Reserve / floor: an ad must clear a minimum eCPM to show at all. Protects revenue (don't sell cheap) and users (don't show junk). Reserves can be dynamic/personalized.
• Quality / relevance & user-value terms: real ad rank is not just bid × pCTR. It blends in ad quality, predicted user satisfaction, and negative signals (hide-rate):

Same two-stage funnel as organic, but eligibility is richer. For each request, retrieve ads that are (a) targeted to this user (audience match), (b) eligible (budget not exhausted, frequency cap not hit, brand-safe for this context), and (c) likely relevant. Methods: inverted-index match on targeting criteria, plus a two-tower ad-retrieval model (user tower × ad tower → ANN index) for relevance-based recall. Output: a few hundred candidate ads.

pCTR model: a large-scale binary classifier. Features: user (history, demographics, interests), ad (creative, advertiser, category, embeddings), context (placement, device, time), and crucially cross features (user×ad affinity). Architectures, in order of how interviewers expect them: logistic regression (the historical baseline, still a great story for calibration/scale), GBDT, Wide&Deep (memorization + generalization), DeepFM / DCN / DLRM (explicit feature crosses + embeddings — the modern default for ads), and increasingly transformer-style sequence models over user history.

Multi-task: predict pCTR, pCVR, and other engagement heads jointly (MMoE/PLE) — they share a backbone and a lot of signal. For oCPM you need pCTR and pCVR to compute eCPM.

This is the single most important ads-ML point. Because eCPM = bid × pCTR, the pCTR is multiplied by real money. If your model says 8% but the truth is 4%, you compute double the eCPM, win auctions you shouldn't, and overcharge the advertiser. So ads models must be calibrated, not just well-ranked.

• Measure it: reliability diagram, expected calibration error (ECE), and the simplest production check — predicted-clicks vs actual-clicks ratio over a window should be ≈ 1.0.
• Fix it: Platt scaling, isotonic regression, or temperature scaling on a held-out set; recalibrate continuously because the marketplace drifts.
• Why training breaks it: negative downsampling (you keep all clicks but sample the huge negative class for tractable training) shifts the base rate, so you must correct the predictions back with the known sampling rate: $p_{\text{corrected}} = \frac{p}{p + (1-p)/w}$ where $w$ is the negative downsample factor.

• Delayed feedback: a click labeled "no conversion" today might convert next week. Naive labeling under-counts positives. Fixes: a delayed-feedback model (jointly model conversion probability and the delay distribution, reweight), or wait a fixed attribution window before finalizing labels (freshness vs correctness tradeoff).
• Selection bias: you only observe conversions among clicks, but pCVR is needed over all impressions. ESMM solves this by modeling pCTR and pCTCVR over the full impression space and getting pCVR via the factorization $p\text{CVR} = p\text{CTCVR} / p\text{CTR}$.

• Scale & sparsity: billions of examples, billions of sparse feature values (ad IDs, user IDs), CTRs of a few percent. Hashing / embedding tables for high-cardinality IDs, negative downsampling + calibration correction, online/continual learning so the model tracks fresh creatives and trends.
• Position bias: ads shown higher get clicked more regardless of relevance. Same fix as organic — model position as a feature you zero-out at serving, or inverse-propensity weight the training clicks — so you learn true relevance, not "was it on top."
• Cold-start: a brand-new ad/creative has no click history → lean on content features (creative embedding, advertiser priors), explore (give it some impressions with an exploration bonus / Thompson sampling) to gather data without tanking revenue.

An advertiser's $1000/day must last all day and reach the right users, not get burned by 11am on cheap morning traffic. A pacing controller throttles the ad's participation to track a target spend curve.

• Throttling / probabilistic pacing: enter only a fraction $p$ of eligible auctions; raise $p$ if under-spending, lower it if over-spending.
• Bid modulation: scale the effective bid by a pacing multiplier $\mu \in (0,1]$ — a PID/feedback controller adjusts $\mu$ so cumulative spend tracks the target. This is the more modern approach (smoother than hard skipping).
• Why it's a control problem: spend is a delayed, noisy signal; over-correct and you oscillate. Frame it as a feedback loop with a setpoint (the spend curve) — that framing alone reads as senior.

• Auto-bidding (value-based / target-CPA / target-ROAS): advertisers say "get me conversions at $20 each" and the system sets the per-auction bid for them — bid = target_CPA × pCVR (bid more when conversion is likely). Most spend on mature platforms is auto-bid; the platform is bidding into its own auction on the advertiser's behalf.
• Bid shading: the industry moved from second-price to first-price auctions (especially in programmatic/RTB) for transparency. But in first-price, bidding your true value means overpaying. Bid shading predicts the minimum winning price and shades your bid down to "just enough to win" — restoring some second-price-like efficiency. It's itself an ML model over auction-landscape data.

• Attribution assigns credit for a conversion across the touchpoints that preceded it: last-click (simple, over-credits the final ad), multi-touch (linear/time-decay/position-based), and data-driven attribution (a model — often Shapley-value-based — distributes credit by marginal contribution).
• Incrementality / lift is the honest question: of the conversions we attributed, how many wouldn't have happened anyway? Measured with holdouts / ghost ads (a control group that's eligible but not shown the ad) or geo experiments. This is the metric that protects advertiser trust — attribution can inflate, lift can't.
• Privacy shift: with ATT/cookie deprecation, deterministic attribution is fading → aggregated/modeled measurement (Aggregated Measurement, conversion modeling) and on-device signals. Worth a sentence to show currency.

• The three-sided balance: advertiser ROI (keeps bids/demand up), user relevance & ad load (keeps attention/supply up), platform revenue (the extract). Optimizing one alone breaks the others — over-monetize and users leave (supply collapses); under-monetize and you leave money on the table.
• Ad load & frequency: cap ads per session, frequency-cap per user/ad, interleave with organic. Treat user retention/session length as a hard guardrail in every ads experiment — block ship if they regress even when revenue rises.
• Brand safety & integrity: don't place ads next to harmful content (advertiser side), and screen ad creatives for policy violations (the content-safety stack from ch14 applies to ads too).
• Auction-pressure / reserve tuning: reserves and ad load are the platform's levers on the revenue/experience tradeoff; they're tuned with long-term holdbacks, not single-session revenue.

• Surface & format: promoted posts in the home/subreddit feed (Reddit's main format), or search ads, or video? Assume feed-native promoted posts unless told otherwise.
• Objective: who's buying what — clicks (traffic), conversions (oCPM), or reach (CPM)? Assume a mix dominated by oCPM/conversion, since that's where the money and the ML depth are.
• Scale: state rough numbers — hundreds of millions of DAU-impressions/day, millions of active ads, <~100ms end-to-end including the auction.
• Constraints: latency budget, advertiser budgets must be respected, user experience guardrails, brand safety, privacy.

Lead with: "I'll optimize Total Value = revenue + advertiser value + user value, with retention as a hard guardrail." That frames the whole design as a marketplace, not a revenue grab.

Walk the funnel diagram (chA1). Retrieval: for each ad request, fetch eligible ads by (a) targeting/audience match (inverted index over targeting predicates), (b) eligibility filter (budget remaining, frequency cap, brand-safe for this context), and (c) a two-tower relevance retrieval (user tower × ad tower → ANN) — narrowing millions of ads to a few hundred. Cache item (ad) embeddings; the user tower runs online.

Score each candidate with a multi-task model (DCN/DLRM-style with embeddings + explicit crosses, MMoE heads for pCTR and pCVR). Emphasize calibration (predicted/actual ≈ 1.0; recalibrate continuously; correct for negative downsampling) and the delayed/biased conversion handling (ESMM + delayed-feedback). Features: user × ad × context + sequence over user history. Mention online/continual learning for fresh creatives.

This is where you stand out. For each candidate compute eCPM = bid × pCTR (× pCVR for oCPM) × 1000, add quality/user-value terms to get AdRank = expected revenue + α·quality − β·p(negative), apply the reserve, run a second-price/VCG auction over the slots, and compute each winner's price. Then blend the winning ads into the organic feed (compare ad eCPM against an organic value floor so ads only appear when they beat the opportunity cost of the slot).

• Pacing: a controller per ad/campaign modulates the effective bid (pacing multiplier μ) to track the spend curve; feedback loop on near-real-time spend.
• Budget bookkeeping: spend must be tracked fast and consistently across the fleet — over-delivery is real money lost. Use a fast counter (Redis-style) with eventual reconciliation; account for the in-flight auctions.
• Serving: feature store (online + offline parity), candidate/embedding caches, the whole request in a tight latency budget; the auction itself is cheap once eCPMs are computed.
• Event pipeline: impressions/clicks/conversions → Kafka → stream features (Flink) + warehouse; log what you ranked with (candidate set, bids, pCTR, position) so training data is reconstructable and you can measure bias.

• A/B with guardrails: revenue up is not a ship criterion if retention/ad-hide regress. Use long-term holdbacks for the revenue/experience tradeoff.
• Incrementality: ghost-ad / holdout measurement so you report causal advertiser value, not just attributed conversions.
• Calibration monitoring: predicted/actual click & conversion ratios on dashboards; alert on drift.
• Marketplace monitoring: auction depth, fill rate, reserve effectiveness, advertiser ROI distribution — the marketplace can degrade silently even when revenue looks fine.

Step 1 — Clarify & objective

"Search over posts, comments, communities, and people. Objective: return results that satisfy the query — measured by click + dwell with no immediate re-query or abandonment — fast. I optimize a relevance proxy but guard against pure engagement-bait results and against high zero-result/abandonment rates. Clarify: is it primarily navigational (find a known sub) or informational (find content about X)? That changes the sparse/dense balance."

Step 2 — Requirements & scale math

Functional: parse query → retrieve candidates → rank → return top-K with pagination + facets (subreddit, time, type). Non-functional: search p99 ~200–300ms (users tolerate a bit more than feed scroll), fresh content searchable within minutes, typo/synonym tolerance. Scale: search QPS is far below feed QPS (a fraction of sessions search) — call it low thousands/sec peak; the cost is the index size (billions of posts/comments) and keeping it fresh.

Step 3 — ML framing

Two-stage: retrieval (cheap, high-recall, get ~hundreds–thousands of candidates) then ranking (expensive, high-precision LTR over the candidates), then optional cross-encoder re-rank on the top ~50–100. Retrieval is hybrid: sparse (BM25 inverted index — exact terms, names, rare tokens, operators) ∪ dense (embedding ANN — semantics, synonyms, paraphrase), fused. Labels: graded relevance from clicks/dwell with position-bias correction (you only see feedback for what you ranked and where).

Step 4 — Data & feature pipeline

Indexing pipeline: new/edited content → CDC/Kafka → tokenize + embed → upsert into the inverted index (Solr/Lucene) and the vector index (HNSW/IVF-PQ). Query-side: query understanding (spell-correct, entity/intent, expansion). Click logs → Kafka → training data with position/propensity logging for unbiased LTR. Per-doc engagement features (score, Wilson, freshness, sub authority) materialized in a feature store.

Step 5 — Model

• Sparse retrieval: BM25 over an inverted index; cheap, exact, great for rare strings and proper nouns.
• Dense retrieval: a bi-encoder (two-tower) embedding of query and doc; ANN search; catches vocabulary mismatch. (Bi-encoder because it's precomputable and scales to retrieval.)
• Fusion: reciprocal-rank fusion or a weighted blend of the two candidate sets.
• LTR ranker: LambdaMART (GBDT, listwise, directly optimizes NDCG) as the battle-tested baseline over relevance + engagement + freshness + authority features.
• Cross-encoder re-rank: query+doc jointly through a transformer on the top ~50–100 — far more accurate, far slower, so only on the short list.

Step 6 — Serving architecture & APIs

 query ─▶ query understanding ─▶ ┌─ BM25 (inverted index) ─┐ ─▶ fuse (RRF) ─▶ LTR ─▶ cross-encoder ─▶ top-K
          (spell, intent,             └─ dense ANN (vector idx) ┘       (~hundreds)  (LambdaMART)   (top ~50)
           expansion)                                                                       + feature store reads
    

Exposed as a search service / GraphQL field; results hydrated through the same DataLoader + subgraph path as the feed. Facets and pagination via cursors.

Step 7 — Scale / freshness / reliability

• Index sharding: shard the inverted + vector indexes by doc; scatter-gather across shards, merge top-K. Replicate shards for read QPS and availability.
• Freshness: near-real-time indexing off CDC/Kafka so new posts are searchable within minutes; segment merges in the background.
• Cross-encoder cost: only on the top ~50–100, batched on GPU — this is the latency-critical stage; degrade by skipping the cross-encoder under load (LTR results only).
• ANN operating point: tune efSearch/nprobe to a recall@K vs p99 target; cache hot/head queries entirely.

Step 8 — Eval, A/B, monitoring

• Offline: NDCG@K, MRR, recall@K for retrieval; on position-bias-corrected labels.
• Online: success rate (click + dwell, no re-query/abandonment), zero-result rate, time-to-first-click; A/B with these as primary.
• Monitoring: retrieval recall, query-latency p99 per stage, index freshness lag, head-query cache hit rate, ANN recall.

Step 1 — Clarify & objective

"Recommend short/long videos to maximize satisfying watch — completion + meaningful watch-time, not just clicks (clickbait inflates clicks, kills satisfaction). Guardrails: not-interested/skip rate, reported content, long-term retention. Clarify surface (feed vs dedicated video tab), cold-start severity, and whether autoplay is on (changes the impression/label definition)."

Step 2 — Requirements & scale math

Functional: rank a personalized video feed; log rich playback telemetry; update on feedback. Non-functional: feed p99 ~150ms, video start fast (separate CDN concern), labels available within minutes. Scale: same DAU order as feed; the new firehose is playback telemetry — play, pause, seek, watch-time-quartiles, buffering — which is denser than clicks (many events per video) → easily the heaviest event stream and the reason logging dominates the discussion.

The three places parity breaks

• Feature-computation skew: the online "watch-time in last 10 min" is a Flink sliding window; if the offline training feature is recomputed by a different Spark query with a different window edge or a different late-data policy, the model trains on values it never sees in prod. Fix: one feature definition, materialized to both stores (ch18), with periodic sampled skew checks comparing served-vs-recomputed values.
• Label-definition skew: "satisfying watch" must mean the same thing in the label pipeline and any online proxy. If offline labels use a 30s completion threshold but the online reward uses raw watch-time, the model optimizes a target you never measured. Pin the label SQL/logic in code and version it.
• Data-availability skew: a feature available at training time (because the lake has the full day) may not be available at serving time (the Flink job is 8s behind). Train on what's actually servable — use point-in-time joins that respect the real serving freshness, not the idealized batch value.

The feedback loop and its bias

Today's served videos generate tomorrow's labels, so the model learns only about what it already showed — a self-reinforcing loop that ossifies the catalog and starves new/long-tail videos. Counter it explicitly: keep an exploration slice (e.g. an ε-greedy or Thompson-sampled fraction of slots), log the propensity (probability the slot was shown) so you can do inverse-propensity-weighted offline eval, and reserve a fresh/cold-start lane in candidate gen. Naming "we only observe outcomes for what we showed, so I log propensities and keep an exploration slice for unbiased eval" is a top-decile signal here.

Step 1 — Clarify & objective

"A feature store is the system that lets many teams define a feature once and consume it consistently in training (historical, point-in-time-correct) and serving (online, sub-10ms), with versioning, tests, lineage, and rollback. The objective is training-serving consistency and developer velocity, with reliability and low serving latency as hard constraints. The #1 thing it exists to kill is train/serve skew."

Step 2 — Requirements & scale math

Functional: register feature definitions; materialize to online + offline stores; serve point-in-time training sets; serve online features at request time; version + test + roll out + roll back + backfill. Non-functional: online read p99 <10ms, offline join correctness (no leakage), high availability of the online path (it's on the critical serving path of every model). Scale: online store handles ranker QPS × features/request — e.g. 50k QPS × 200 features = 10M feature reads/sec, so reads are batched multi-gets and the store is sharded Redis/Cassandra.

Train/serve skew is when a feature is computed differently (or with different data availability) in training vs serving — the silent #1 RecSys model killer. The two mechanisms that prevent it:

• Shared definition + shared materialization code: the same logic computes the feature for the online store and the offline store. No re-implementing "user 7-day click count" twice.
• Point-in-time (as-of) joins for training: when building a training row for an event at time T, join feature values as they were at T, never their current value — otherwise you leak the future into the label. The offline store keeps a full history with valid-from timestamps so an as-of join is exact.

This is the difference between a model that looks great offline and dies in production (leakage + skew) and one that ships.

19.1 Idempotency

What: processing the same event twice produces the same state. Why it's senior: distributed systems retry; at-least-once is the default; so correctness requires idempotent consumers (dedupe by event_id, upsert by natural key, or compare-and-set). Survive the follow-up: "I'd make the consumer idempotent by upserting the per-user/per-window aggregate keyed by event_id, so a replay is a no-op, rather than claiming end-to-end exactly-once which is usually a myth across an external store." (ch2.9)

19.2 Backpressure

What: when a downstream can't keep up, the system slows or buffers gracefully instead of dropping data or cascading failure. Why senior: shows you think about overload, not just the happy path. Say: "Kafka is the backpressure — a slow consumer accrues lag on a durable log; we alert and autoscale on lag, and the read path degrades to bounded-staleness features rather than blocking." (ch2.8)

19.3 Graceful degradation

What: the system gives a worse-but-working answer when a dependency fails, never a blank/error. Why senior: availability is a product feature; "the feed must never be empty." Say: "If the ML ranker times out, fall back to hot-score ordering of cached candidates; if a GraphQL subgraph is down, return a partial response with that field null — degraded, never broken."

19.4 Shadow traffic

What: run a new model/system on live traffic, score it, but don't serve its output — compare against production. Why senior: de-risks launches with real distributions before any user is affected. Say: "Shadow the new ranker on live traffic, compare prediction distributions and latency, then canary 1%, then ramp — with a kill switch the whole way."

19.5 Training-serving skew

What: a feature computed differently online vs offline → the model sees different inputs in prod than in training. Why senior: it's the #1 silent killer of RecSys models and only experienced people name it unprompted. Say: "One feature definition materialized to both the online and offline stores, plus skew tests sampling served-vs-recomputed values, so training-time values equal serving-time values." (ch6.3, ch18)

19.6 Kill switches & rollback

What: every model/risky feature can be instantly disabled and reverted to a known-good version without a full redeploy. Why senior: shows operational maturity — you plan for being wrong. Say: "Every model is an immutable versioned artifact behind a pointer; rollback is a repoint, and there's a per-feature kill switch so a bad feature drops out of serving without redeploying every model."

19.7 Online/offline metric parity

What: the offline metric you optimize should move the online metric you care about; verify the link, distrust offline-only wins. Why senior: guards against shipping models that win on NDCG but lose on retention. Say: "I check that offline AUC/NDCG gains correlate with the online guardrails in past A/Bs; if they decouple, I trust the online result and re-examine the offline proxy. And I never ship on an engagement lift if a health guardrail regresses."

19.8 Cost

What: name the dominant cost (GPU fleet, embedding memory, cross-region egress, always-on stream compute) and the lever to control it. Why senior: Staff engineers own budgets, not just latency. Say: "The cost driver here is the GPU ranker fleet sized for peak p99; I cap it with dynamic batching, a cascade that keeps the expensive model off the obvious cases, and caching the feed so most reads never hit the ranker. Stream compute is always-on, so I'd only stream the features that genuinely need second-level freshness and batch the rest."

Step 1 — Clarify & objective

"Given a user question, return a helpful, grounded answer that synthesizes across Reddit threads and cites the specific posts/comments it used. The true objective is helpful + faithful + safe, not 'fluent.' I optimize answer helpfulness (did it resolve the question, thumbs-up, no immediate re-ask) under hard constraints: faithfulness (every claim supported by a cited source), attribution (Reddit's value is community knowledge — cite the humans), and safety (no harmful/medical/legal mis-advice, no surfacing of toxic or removed content). Clarify scope: is it answering from the whole corpus or in-context within a subreddit/thread? Is it conversational (follow-ups) or single-shot? Both change retrieval and memory."

Step 2 — Requirements & scale math

Functional: parse question → retrieve relevant passages → rerank → generate a cited answer → safety-gate → return with source links; support follow-ups. Non-functional: end-to-end latency is dominated by generation — target first-token < ~1s, full answer ~2–5s (users tolerate far more than a 150ms feed scroll, but it must feel responsive, so stream tokens). Freshness: new threads answerable within minutes–hours. Cost: LLM generation is the dominant cost — every design lever (retrieve-less, cache, smaller model, distill) trades quality vs $/query.

Scale: question QPS is far below feed QPS — call it low hundreds to low thousands/sec. The cost isn't QPS, it's (a) the index over billions of posts + comments and keeping it fresh, and (b) the per-query LLM tokens. Back-of-envelope: 1k QPS × (say) 2k input tokens (retrieved context) + 400 output tokens → ~2.4M tokens/sec of generation → this is why context budget and caching are load-bearing cost decisions, and why you don't stuff 50 passages into the prompt.

Step 3 — ML framing (the RAG pipeline)

  question ─▶ query understanding ─▶ hybrid retrieval ─▶ rerank ─▶ context build ─▶ generator (LLM) ─▶ grounding/cite check ─▶ safety gate ─▶ answer
              (intent, decompose,        sparse(BM25)            cross-       (dedup, pack,    (grounded,        (verify each claim   (toxicity, policy,   + citations
               rewrite, expand,          ∪ dense(ANN)            encoder      budget tokens,    cite-constrained   maps to a source;     PII, no removed     (links to
               subreddit hints)          → top ~100              top ~10-20)  per-source IDs)   decoding)          drop unsupported)     content)            threads)
    

It's the search funnel (ch16) with a generation head bolted on, plus two RAG-specific stages: context construction (what passages, in what order, within a token budget, tagged with stable source IDs so citations are machinable) and grounding/citation verification (post-hoc check that every claim is supported by a retrieved source; drop or flag unsupported claims). The generator is one stage of eight — say that explicitly so you're not seen as "just call an LLM."

Step 4 — Data & indexing pipeline

• Chunking: Reddit content is threaded — a question's answer often spans a post + several comments. Chunk at a passage granularity (a comment, or a post + its top-Wilson comments) with enough context to be self-contained, and keep a stable source ID (permalink, thread id, comment id) on every chunk so citations resolve to a real, linkable thread. Over-chunking loses context; under-chunking blows the token budget.
• Embedding + dual index: embed each chunk → upsert into a vector index (HNSW/IVF-PQ, ch5.6) and an inverted index (BM25, for exact terms/names/error codes). New/edited content flows via CDC/Kafka (ch5.7) so the index is fresh within minutes.
• Metadata to index alongside: subreddit, score/Wilson, recency, author trust, removed/locked/NSFW flags — used both as retrieval filters and rerank features, and to exclude removed/unsafe content from ever being a source.
• Training data: for the reranker — graded relevance from clicks on cited sources, thumbs-up/down on answers, and (offline) human-labeled query↔passage relevance. For eval — a held-out set of questions with reference answers and known supporting sources.

Step 5 — Model (each stage)

Query understanding

Detect intent (factual / opinion / how-to / current-event), rewrite conversational follow-ups into standalone queries (resolve "what about the cheaper one?" against history), optionally decompose a multi-part question into sub-queries, and expand (synonyms, entities). For conversational mode this rewrite step is what makes follow-ups work — name it.

Retrieval (hybrid)

Sparse BM25 (exact subreddit names, product names, error strings) ∪ dense bi-encoder ANN (semantic match, paraphrase), fused with reciprocal-rank fusion → top ~100 passages. Hybrid because dense models miss exact rare strings and sparse misses vocabulary mismatch (ch16.7). Filter out removed/NSFW/low-quality at retrieval.

Reranking

A cross-encoder (query+passage jointly through a transformer) reranks the top ~100 → top ~10–20 (ch16: retrieve cheap, rerank precise). Blend in Reddit-native signals: Wilson score, subreddit authority, recency, author trust — a highly-upvoted answer in a relevant sub is better grounding than an obscure low-score comment.

Generator (the LLM)

• Grounded, citation-constrained generation: the prompt packs the reranked passages each tagged with a source ID; the instruction is to answer only from the provided sources and to attach a citation ID to each claim. This is the core anti-hallucination mechanism — the model is told its evidence and told to cite it.
• Context budget: pack the highest-value ~10–20 passages within the token budget (dedup near-duplicate comments; order by relevance; truncate tails). More context is not free — it costs latency, money, and can dilute attention ("lost in the middle").
• Abstain path: if retrieval returns nothing relevant (low scores), the generator should say "I couldn't find a good answer on Reddit" rather than hallucinate. An explicit no-answer is correct behavior, not failure.
• Model choice: a strong instruction-tuned LLM; consider a smaller distilled model for cost if quality holds. The model is swappable — the grounding/citation contract around it is the durable design.

Grounding / citation verification

Post-generation, verify that each sentence/claim is entailed by at least one cited source (a lightweight NLI/entailment check or a verifier model). Drop or flag claims with no support; ensure every citation ID resolves to a real passage that actually backs the claim. This is what turns "an LLM with sources in the prompt" into a faithful system.

Step 6 — Serving architecture & APIs

 question ─▶ GraphQL/answers svc ─▶ query understand ─▶ ┌ BM25 (Solr/Lucene) ┐ ─▶ RRF fuse ─▶ cross-encoder rerank ─▶ context build
                    │                                            └ dense ANN (vector idx)┘     (top~100)        (GPU, top~10-20)        (token budget,
                    │                                                                                                                    source IDs)
                    ▼                                                                                                                         │
            conversation memory                                                                                                              ▼
            (rewrite follow-ups)                          answer ◀─ safety gate ◀─ grounding/cite verify ◀─ LLM generator (streamed tokens)
                                                          + cited                (NLI entailment,            (grounded, cite-              GPU, dynamic-batched,
                                                          thread links            drop unsupported)           constrained decoding)         KV-cache, semantic cache
    

Exposed as an Answers service / GraphQL field; sources hydrate to real thread links through the same DataLoader + subgraph path (ch4). Stream tokens to the client so perceived latency is first-token, not full-answer. Each component is independently scalable: retrieval and rerank are GPU/CPU index reads; generation is the GPU-bound, dynamically-batched, KV-cache-heavy stage that dominates cost.

Step 7 — Scale / latency / cost / freshness / reliability

• Latency budget (generation dominates): ~30ms query understanding + ~40ms hybrid retrieval + ~60ms cross-encoder rerank + ~300ms-to-first-token + streamed generation (~2–4s full). So stream, and parallelize retrieval branches. Cross-encoder and verification are the droppable-under-load stages (degrade to LTR-only ranking, skip the verifier with a confidence flag).
• Caching (huge cost lever): semantic cache on (normalized question → answer) for popular/duplicate questions; prefix/KV cache for the shared system prompt + instructions; cache retrieval results for repeated queries. A high cache hit rate on head questions can cut generation cost by a large fraction.
• Cost levers: retrieve/pack fewer, better passages (rerank precision > raw context size); use a smaller/distilled generator where quality holds; cap output length; semantic-cache head queries. Quote: "generation is the dominant $/query, so I spend on rerank precision to shrink the context rather than stuffing the prompt."
• Freshness: near-real-time indexing off CDC/Kafka so new threads are retrievable within minutes — critical for "what's happening with X right now" questions. For genuinely breaking topics, boost recency in rerank.
• Reliability / graceful degradation: if the generator is down or times out, fall back to showing the top retrieved threads (an extractive/search result) — degraded but never blank, and it's still grounded. If retrieval is empty, abstain honestly. Kill switch on the generator to revert to search-results mode.

Step 8 — Eval, A/B, monitoring, safety

• Faithfulness / groundedness: the headline metric — fraction of answer claims supported by cited sources (measured by an NLI/entailment judge or human raters). A fluent-but-unsupported answer is a failure.
• Citation accuracy: do the cited sources actually back the claims, and do citation IDs resolve to real threads? Precision/recall of citations.
• Retrieval quality: recall@K and NDCG of the retriever/reranker on labeled query↔passage relevance — the upstream ceiling on answer quality.
• Answer helpfulness: offline LLM-as-judge + human eval; online: thumbs-up rate, no-immediate-re-ask, dwell on the answer, click-through to cited threads.
• Safety: harmful-advice rate (medical/legal/financial), toxicity, surfacing of removed/NSFW content as a source, PII leakage — gated and audited; A/B guardrails block ship on any safety regression.
• Online A/B + monitoring: helpfulness/thumbs-up as primary; faithfulness, safety, and latency as guardrails. Monitor retrieval recall, index freshness lag, abstain rate, hallucination/unsupported-claim rate, cache hit rate, p99 first-token, and $/query.

1. Problem framing

Predict P(click | user, ad, context) for a candidate ad in a slot. It drives the auction: rank = bid × pCTR and the price depends on it, so this is a calibrated probability problem, not just a ranking problem (ch15). Online scoring, per impression candidate.

2. Label definition (where points are won)

• Positive = a click on the ad — but define it carefully: filter accidental clicks (immediate bounce-back), bot/invalid traffic, and double-counts (dedupe by impression_id). A "click" that bounces in 1s isn't a real positive.
• Attribution window: click must be tied to a specific impression; log impression_id on the click so the join is exact.
• Negatives: impressions shown but not clicked — but beware position/examination bias: a non-click at the bottom of the page may be "not seen," not "not interesting." Either model position explicitly (and zero it at serve) or correct with examination propensities.

3. Data & sampling

• Extreme imbalance: CTR is often ~1–5%, so positives are rare. Use negative downsampling (keep all positives, sample negatives) for tractable training, then recalibrate the predicted probability back to the true prior (downsampling inflates predicted rates — a known correction). This is a classic ad-modeling detail interviewers probe.
• Time-based split: train on past, validate on future — random splits leak temporal patterns and overstate offline metrics.

4. Features

• User: interest embeddings, historical CTR, recency/frequency of ad engagement, demographics/context.
• Ad/creative: creative embeddings, ad category, historical ad CTR (with smoothing for new ads), advertiser.
• Context: placement/slot, device, time, the surrounding organic content.
• Cross features: user-interest × ad-category, the most predictive and the reason feature crosses / DCN-style models matter for ads.
• Leakage check: "ad's CTR" must be computed as of the impression (point-in-time), never including the current click; new-ad CTR needs a smoothed prior, not a leaked future average.

5. Model family + why

Two strong choices, and I'd name the tradeoff: GBDT (XGBoost/LightGBM) — excellent on tabular, handles nonlinearity and missing values, strong baseline, fast to iterate; or a DNN with embeddings + explicit feature crosses (Wide&Deep / DeepFM / DCN) — better at high-cardinality sparse IDs (user/ad/advertiser) and learned crosses, scales to huge data, and supports multi-task (CTR + CVR). For ads at scale with massive sparse ID spaces, the industry standard is the deep cross-network family; I'd start with GBDT as a baseline and move to a DCN/DeepFM if the sparse-ID lift justifies the serving complexity. Logistic regression with hashed crosses is the classic, still-respectable, ultra-fast fallback.

6. Loss & calibration (load-bearing for ads)

Train with log-loss / binary cross-entropy (proper scoring rule → encourages calibrated probabilities). Then add an explicit calibration layer — isotonic regression or Platt scaling — because (a) negative downsampling biased the probabilities, and (b) the auction prices on the numeric probability. A pCTR that's 2× too high overspends real advertiser money. Calibration is non-negotiable here, unlike organic ranking where only order matters.

7. Evaluation

• Offline: AUC / PR-AUC for ranking quality, log-loss and a reliability/calibration curve + expected calibration error for probability quality. Report both — a high-AUC, badly-calibrated model still mis-prices.
• Online experiment: A/B with revenue/RPM as primary, advertiser ROI and user guardrails (ad-hide rate, session length, retention) as guardrails. Block ship if user guardrails regress even when revenue rises.

8. Failure modes

• Miscalibration drift → overspend; monitor predicted-vs-actual CTR continuously, recalibrate on drift.
• Position bias → model just learns "top = clicked"; correct with position feature / propensities.
• Cold-start ads → no history; smoothed priors + exploration (Thompson sampling) so you learn pCTR without burning budget.
• Feedback loop → you only see clicks on ads you showed; log propensities, keep exploration, evaluate with IPW.
• Leakage → any feature peeking at the label or computed post-impression silently inflates offline metrics and collapses online.

1. Problem framing

Predict expected dwell-time (or watch-time for video) on a feed item, used as a satisfaction signal blended into the ranker (ch7/ch17). Unlike click (binary), dwell is a continuous, heavy-tailed, censored target — which drives every downstream choice.

2. Label definition (the crux)

• Dwell-time is heavy-tailed (most items get a few seconds, a few get minutes) → model log(dwell) or bucketize, don't regress raw seconds with MSE (the tail dominates the loss).
• Censoring: a user who closes the app mid-item gives a lower bound, not a true dwell — handle with survival/Tobit-style treatment or cap-and-flag, not by treating the truncated value as truth.
• "Satisfying" vs raw dwell: raw dwell rewards long items and "couldn't find the exit" passive watching. Define the target as meaningful dwell (past a threshold, normalized by item length, paired with positive feedback), and treat fast-skip as a strong negative. Same lesson as ch17.6.
• Position/autoplay bias: dwell is confounded by position and by autoplay; condition on or correct for them.

3–5. Data, features, model

• Features: user historical dwell by topic, item length/type, content embeddings, context (time, device, autoplay flag), position. Cross: user-topic × item-topic.
• Model: a regression head on log-dwell, often a head on the shared multi-task ranker (P(click), P(complete), E[dwell], P(skip)) so it shares representations and the heads trade off in the blend. Could be GBDT for a standalone baseline; DNN multi-task in the production ranker.
• Two-stage option: P(dwell > 0) × E[dwell | dwell > 0] to handle the spike at zero (didn't engage) plus the continuous tail — cleaner than one regression over a zero-inflated target.

6. Loss & calibration

Regression loss on the transformed target (e.g. MSE on log-dwell, Huber for tail robustness, or a quantile/pinball loss if you want the distribution). Calibrate if E[dwell] is blended numerically with classification heads into one score — an overpredicting dwell head silently dominates the blend.

7. Evaluation

Offline: regression error on the transformed scale (don't report raw-second RMSE — the tail makes it meaningless), correlation/ranking metrics (does higher predicted dwell rank items the way actual dwell does), and calibration of E[dwell] by segment (short vs long items — regression heads quietly mispredict on the tails). Online: A/B on satisfying-watch / session length with negative-feedback (skip/not-interested) guardrails — never ship a dwell lift that raises the skip rate.

8. Failure modes

• Length bias → predicting "long item = high dwell"; normalize by length, optimize completion not raw seconds.
• Censoring ignored → systematically under-predicting dwell for engaged sessions cut short.
• Tail misprediction → MSE on raw seconds chases outliers; use log/Huber/quantile and monitor by segment.
• Optimizing the wrong thing → raw dwell rewards passive/clickbait watching; pair with explicit positive feedback and skip as a negative.

Clarify & requirements

"A chatroom per subreddit (or a thread): many users send messages, everyone in the room sees them in near-real-time, with scrollback history, presence ('who's online'), and moderation. Clarify: room size (a small private room vs an r/popular live thread with 100k concurrent — wildly different fan-out), ordering guarantees, and retention. I'll assume rooms can get large, so fan-out and hot-room handling are the interesting part."

• Functional: join/leave a room; send a message; receive others' messages live; load history (paginated); presence; moderation (delete/ban/rate-limit).
• Non-functional: low delivery latency (sub-second), ordered-within-room, durable history, scale to large rooms, graceful under hot rooms.

Architecture

  clients ──(WebSocket)──▶ gateway / connection servers ──▶ Kafka/Redis pub-sub (per-room topic/channel) ──▶ fan-out to subscribers
     │  send msg                  (hold the WS connections;            │  (decouples senders from
     │                            map room → connected sockets)        │   the N connection servers)
     │                                      │                          ▼
     │                                      │                 chat service ─▶ persist to Cassandra
     │                                      │                 (validate, rate-limit,   (messages by (room_id, bucket),
     │                                      ▼                  moderation hook)         clustering by ts → history scan)
     │                          presence in Redis (room → online set, TTL heartbeat)
     ▼
  history read ──▶ chat service ──▶ Cassandra range-scan (paginate by cursor)
    

The key idea: clients hold a WebSocket to a stateful connection server; a published message goes through a pub-sub fabric (Redis pub-sub or a Kafka topic per room) so it reaches every connection server that has subscribers in that room, not just the sender's. Persistence is decoupled — write to Cassandra keyed by (room_id, time_bucket) so history is a single-partition range-scan and a hot megaroom doesn't make one unbounded partition (time-bucketing caps partition size — ch6.7 failure mode).

Message + API shape (the "coding" part)

// message
{ id: uuid, room_id, author_id, body, ts, seq }   // seq = per-room monotonic for ordering/dedup

// Cassandra table (history = single-partition range scan)
CREATE TABLE messages (
  room_id   uuid,
  bucket    int,                 -- e.g. hour bucket, caps partition size
  ts        timeuuid,
  msg_id    uuid,
  author_id uuid,
  body      text,
  PRIMARY KEY ((room_id, bucket), ts)
) WITH CLUSTERING ORDER BY (ts DESC);

// WS protocol (client ↔ gateway)
-> JOIN  {room_id}
-> SEND  {room_id, body, client_msg_id}     // client_msg_id for idempotent send/dedupe
<- MSG   {id, author, body, ts, seq}         // broadcast to room
<- ACK   {client_msg_id, id, seq}           // confirm the sender's own send
<- PRESENCE {room_id, online_count}

// history (REST/GraphQL)
GET /rooms/{id}/messages?before={cursor}&limit=50   // cursor = last seen ts/seq

Ordering & dedup: assign a per-room monotonic seq at the chat service (or use a timeuuid); clients dedupe on id and reconcile their optimistic local echo via the ACK's client_msg_id. Delivery: at-least-once over the pub-sub + idempotent client render (dedupe by id) — the same "at-least-once + idempotent" pattern from ch2.9.

Scale, hot rooms, moderation, reliability

• Hot room (100k concurrent, e.g. a live r/popular thread): fan-out to 100k sockets per message is the bottleneck. Spread connections across many connection servers, fan-out via the pub-sub fabric so each server pushes to its local subscribers, and batch/coalesce rapid messages. For extreme rooms, sample/aggregate (don't deliver every message to every client; this is the "live thread" degradation, like Twitch chat).
• Presence: Redis set per room with heartbeat TTL; show approximate counts for huge rooms (exact presence at 100k is wasteful).
• Moderation: messages pass a rate-limiter (token bucket — a confirmed Reddit coding topic) and a fast safety check (reuse ch12/ch14's Lua-rules + classifier idea) before broadcast; mods can delete (tombstone) and ban.
• Reliability: on reconnect, the client replays from its last-seen seq/cursor to backfill missed messages from Cassandra — so a dropped WebSocket doesn't lose history. Connection servers are stateless beyond the live socket map, so one dying just drops connections that reconnect elsewhere.

1. Watch-next / video recommendation verbatim staff ML-SD (ch17)

2. Reddit Answers / RAG over Reddit the offer team (ch20)

3. Home feed marquee (ch7)

4. Ad-click modeling ML case study, modeling-only (ch21)

5. Notifications / push confirmed 4-stage (ch9)

6. Search ranking reported (ch16)

7. Content safety / spam / toxicity confirmed Kafka+Flink+Lua (ch14)

8. Feature store darkinterview #19, production-side (ch18)

9. Comment ranking / "best" sort + nested threads confirmed Wilson (ch8)

10. Subreddit / community recommendations cold-start-heavy (ch10)