ML systems — from zero to expert

A traditional web service behaves exactly the way its code says. Change nothing, get the same output every time. An ML system breaks that contract in four specific ways:

Every production ML system — from a spam classifier to a 100B-parameter language model — contains these eight components. Memorize this skeleton. The rest of the course fills each box with depth.

The same model artifact is used in both phases, but the surrounding workload is so different that they often run on separate infrastructure, separate teams, and separate cost budgets. Confusing them in an interview is a red flag.

A useful way to keep these straight: training is a batch job that produces an artifact; inference is a service that consumes it. Everything else follows from this distinction.

Real systems use different names for these components. Here is how to translate.

Trace a single event through the system to see how all eight boxes interact. Suppose a user clicks on a product in an e-commerce feed.

1. Event logged. The click is written to a Kafka topic and archived to the raw data lake within seconds.
2. Feature pipeline fires. A streaming job (Flink) consumes the Kafka event, computes derived features ("user clicked on category=Electronics in the last 10 minutes"), and writes them to the online store. A batch job (Spark) backfills the same features to the offline store once per hour.
3. Training reads offline store. Nightly, a training job reads a point-in-time join of historical events and their features, trains a ranking model, evaluates it on a held-out set, and registers the artifact in the model registry with metrics attached.
4. Serving reads online store. When the user's next page loads, the serving fleet fetches the user's fresh features from the online store (<5ms), runs the registered model, and returns a ranked list of products.
5. Monitoring watches both. The monitoring system checks that the feature distributions match training-time distributions, that model scores are in range, and that downstream metrics (CTR) are healthy. If the user eventually buys a product, that ground-truth label flows back through the feedback loop and becomes new training data.

This end-to-end trace is the feedback loop. Systems without it accumulate technical debt quietly: the world changes, labels accumulate, but the model never learns.

In traditional software, failure is loud. An exception propagates, a test fails, a dashboard goes red. In ML systems, the most dangerous failures are invisible. Here are the three archetypes.

Setup: a social platform allows users to post comments on any item. Approximately 1 billion comments are posted per day. Roughly 2% are spam (20 million spammy comments/day). The goal is to suppress spam before it is shown to other users, with a maximum end-to-end latency of 200ms per comment and a target precision of 95% (at most 5% of suppressed comments are legitimate) and recall of 85% (catch at least 85% of all spam).

This example is deliberately representative: it has high data volume, label delay, a feedback loop, a two-class imbalanced problem, and a latency requirement tight enough to force architectural choices. Virtually every issue you will encounter in real ML systems appears somewhere in this lifecycle.

Every comment posted generates a log event containing: comment text, author user ID, item ID, timestamp, device fingerprint, IP address, and whether the author's account is new (<7 days old). These events are written to a Kafka topic and archived to a data lake (say, S3) in Parquet files partitioned by dt=YYYY-MM-DD/hr=HH.

At 1B comments/day, with an average serialized event size of ~500 bytes, that is roughly 500GB of raw data per day. A year of data is ~180TB. This shapes every downstream design choice: you will not be loading this into Pandas for training; you will use distributed processing (Spark).

Two label sources: (1) User reports — users flag comments as spam. Signal is noisy (users disagree, some abuse the report button) and delayed (a comment may not be reported for hours or days, if ever). (2) Human review queue — a fraction of reported comments go to a team of reviewers who apply a gold-standard binary label (spam / not-spam).

On a given day, about 500,000 comments are reviewed. That is 0.05% of daily volume. The rest — 99.95% — are unlabeled. This creates two problems:

Concrete fix for label delay: when building a training dataset, only include comments where the review decision has been finalized for at least 24 hours. A comment posted on day T is eligible for training on day T+1 at the earliest. The feature values used for that training example are those available at time T (prediction time), not T+1.

A training example consists of: (a) features available at comment-post time, and (b) the ground-truth label. Constructing this correctly requires a point-in-time join.

Consider a feature: "number of spam comments by this author in the last 7 days". This is a powerful feature — serial spammers post repeatedly. The danger: if you compute this feature using all historical data (not restricting to before the prediction timestamp), you will accidentally include future spam posts by the same author, inflating the feature and creating leakage.

Here is the wrong vs right join on a 4-row dataset:

The wrong join gives the model perfect information about how many spam posts the author will eventually make — information that is unavailable at real prediction time. The model appears to have learned a great feature; it has actually memorized the future. Offline AUC: spectacular. Online performance: terrible.

Why this specific failure is so common: the SQL that produces training data often joins two tables on author_id without any timestamp filter. Adding AND spam_log.created_at < comment.created_at fixes it, but this is easy to forget and the resulting bug is invisible until you go to production.

With a correctly built dataset (say, 10 million labeled examples, 30% spam after reweighting), you train a gradient-boosted tree (e.g., XGBoost) or a text classifier depending on whether you emphasize structured features (author behavior, IP reputation, device fingerprint) or text features (comment content).

Concrete features used (with motivation):

Training run: 10M examples, XGBoost with 500 trees, depth 6, on a single 32-core CPU machine. Training time: ~4 minutes. Offline metrics on a holdout set from the next week's data: AUC 0.96, precision@0.5-threshold 0.94, recall 0.87. Both goals met.

Offline evaluation is the first gate before any real traffic sees the model. For this spam classifier, evaluate on a held-out week of data with the following metrics:

The model passes offline evaluation. It is registered in the model registry with all metrics attached, the training data version, and a link to the evaluation report. It is now in status CANDIDATE.

Shadow mode (also called shadow deployment or dark launch) routes a copy of every live request to the new model, runs inference, and logs the result but does not act on it. The current production model still makes all actual suppression decisions.

What shadow mode reveals that offline evaluation cannot:

Shadow mode runs for 48 hours. The feature distributions look healthy. P99 latency is 140ms — within budget. Score distribution matches offline evaluation. The model is promoted to status SHADOW_PASSED.

An A/B test exposes the new model to a random slice of traffic while the old model serves the rest. The new model's decisions are now live: comments it suppresses are actually suppressed.

Design choices for this A/B test:

Results after 14 days: spam-report rate down 18% in treatment, complaint rate flat, latency flat. Statistically significant (p < 0.01). The model is approved for full rollout. Status: PROMOTED.

The model is now serving 100% of traffic. This is not the end of the story; it is the beginning of a new phase.

What gets monitored and why:

Three months after launch, the monitoring dashboard shows: spam-report rate up 12%, suppression rate unchanged. Spammers have adapted their language; the model is missing new spam patterns that did not exist in the training set.

This is concept drift: the relationship between features and the label has changed. The model was trained on comments where URL-heavy, short text predicted spam. Spammers now post long, URL-free text with embedded keywords — the model has never seen this pattern and scores it low.

Detection: the score distribution for user-reported spam that was NOT suppressed is plotted. This is the model's false-negative population. Two months ago, their scores clustered around 0.3 (model was uncertain). Now they cluster around 0.1 — the model is confidently wrong about new spam. This is the signal for retraining.

The team establishes a retrain cadence. The right cadence depends on drift rate and label availability:

For this spam classifier, a weekly retrain with trigger-based emergency retrain is implemented. The weekly job is automated: Airflow DAG → Spark dataset build → XGBoost training → evaluation gate (must meet precision/recall targets) → auto-register in model registry → shadow mode for 6 hours → auto-promote if shadow looks clean. Human review is only required if the evaluation gate fails.

Imagine you have three services that produce events (a web server, a mobile app, and an ad system) and four services that want to consume those events (a feature pipeline, a fraud detector, an analytics DB, and a real-time dashboard). Without a message bus, every producer must know about every consumer and push data directly. That's 3 × 4 = 12 direct connections, each with its own retry logic, backpressure handling, and failure mode. Add one more consumer: you now need 15 connections, and every producer must be updated. This is brittle.

Kafka solves this with a single abstraction: an append-only distributed log. Producers write to the log; consumers read from wherever they left off. Nobody needs to know who else is in the room.

Suppose a topic user-clicks has 3 partitions (P0, P1, P2). Messages are routed to partitions by hashing a key — here, the user_id — so all clicks from user 42 always land in the same partition (guaranteeing order per user). Two consumers (C0, C1) join consumer group feature-pipeline.

Topic: user-clicks  (3 partitions)

P0: [offset 0: uid=7,  clicked=item_A]
    [offset 1: uid=19, clicked=item_C]
    [offset 2: uid=7,  clicked=item_D]   <-- C0 reads P0

P1: [offset 0: uid=42, clicked=item_B]
    [offset 1: uid=42, clicked=item_E]   <-- C0 reads P1

P2: [offset 0: uid=31, clicked=item_F]
    [offset 1: uid=55, clicked=item_G]   <-- C1 reads P2

Consumer group "feature-pipeline":
  C0 owns P0, P1  |  C1 owns P2

C1 processes 1 partition; C0 processes 2. To balance load, add a third consumer C2: Kafka rebalances automatically — C0, C1, C2 each own one partition. Add a fourth consumer and one sits idle (can't split one partition further). Rule: max useful parallelism = number of partitions.

Meanwhile, a completely separate consumer group fraud-detection reads the same topic from offset 0 independently. Kafka does not care — it just serves sequential reads from its log.

Once events land in Kafka (or a data lake), you must process them. The two primary models are batch and streaming, with a micro-batch middle ground. This decision affects latency, cost, correctness guarantees, and operational complexity — so interviewers probe it constantly.

Batch processing (Apache Spark): read a bounded dataset, run a computation, write results. Spark breaks the dataset into partitions, distributes them across a cluster, applies your transformation in parallel, and aggregates results. A typical nightly job: read all events from the last 24 hours, compute feature aggregates (7-day user click counts, 30-day purchase averages), write to the offline feature store. Throughput is very high; latency is high (hours). Cost is lower per byte because compute can be spot/preemptible.

Streaming processing (Apache Flink): process each event (or micro-window) as it arrives. Flink maintains persistent operator state (e.g., rolling counts), reacts to each Kafka message in milliseconds, and continuously updates the online feature store. Latency is low (seconds); cost is higher because you need always-on compute. Streaming is harder to reason about: what happens when an event arrives late? What is "count of clicks in the last hour" when the pipeline restarts?

Micro-batch is the pragmatic middle ground: Spark Structured Streaming triggers a mini-batch every N seconds (configurable). You get streaming semantics (continuous query, stateful operators) with batch execution (full Spark optimizations). Latency is typically 5–60 seconds — fine for most ML features, not fine for fraud detection or live auctions.

These three terms describe where and how data is stored at rest. Mixing them up in an interview signals unfamiliarity with data infrastructure.

For ML specifically: raw logs land in the lake first (cheap, everything is kept). A batch pipeline reads the lake, processes events, and writes structured feature tables to the warehouse (or lakehouse). The offline feature store for training is typically a warehouse table or Iceberg/Delta table. The online feature store is a separate low-latency KV store (Redis, DynamoDB) — the warehouse is too slow for serving.

Parquet (and ORC) store data column-by-column rather than row-by-row. This sounds like a detail but it is transformative for ML feature computation.

Suppose you have a table with 1 billion rows and 200 columns — common for user event logs. You want to compute the mean click rate (column 7) for a specific user segment. With row-oriented storage (like a CSV or PostgreSQL heap), the database reads every byte of all 200 columns for all 1 billion rows, even though you care about one column. At 200 bytes/row, that's 200 GB of I/O to get 1 GB of useful data. Selectivity: 0.5%.

With Parquet, column 7 is stored contiguously. The query reads only that column: 1 billion × 4 bytes = 4 GB, plus it skips entirely rows that fail the filter (row group statistics let Parquet skip entire 128MB chunks where min > threshold). Practical speedup: 10–100×.

Additionally, each column compresses far better than mixed rows — values within a column are similar (all timestamps, all float click rates), so SNAPPY or ZSTD achieves 5–10× compression. That 1 GB column becomes ~150 MB on disk.

This is one of the most commonly misunderstood concepts in stream processing — and a favorite interview probe for anyone touching real-time pipelines.

Why the distinction matters: suppose you want to count clicks per minute. Using processing time is easy but wrong — a batch of events delayed by 30 seconds (network blip, mobile app buffering) will inflate the next minute's count and undercount the previous one. Using event time gives the correct count per minute but introduces a new problem: when is a window complete?

Worked example with timestamps:

Event  |  event_time  |  processing_time  |  Window (event_time)
-------+-------------+-------------------+----------------------
E1     |  09:00:10   |  09:00:11         |  09:00:00–09:01:00
E2     |  09:00:45   |  09:00:46         |  09:00:00–09:01:00
E3     |  09:00:58   |  09:01:32         |  09:00:00–09:01:00  <-- LATE: arrives 34s late
E4     |  09:01:15   |  09:01:16         |  09:01:00–09:02:00

E3 has event_time 09:00:58 — it belongs to the 09:00 window — but it arrives at 09:01:32, after the processor has already seen E4 from the 09:01 window. The processor cannot wait forever. Watermarks solve this.

A watermark is the processor's estimate of the maximum event_time it has seen, minus a configured lag (e.g., 30 seconds). When processing_time = 09:01:32 and the max event_time seen so far is 09:01:15, the watermark is 09:01:15 − 00:00:30 = 09:00:45. This means: "I believe all events with event_time ≤ 09:00:45 have now arrived." Windows are closed when the watermark passes their end time.

E3 (event_time 09:00:58) arrives after the watermark has already passed 09:00:58. It is a late event. Options: drop it, emit a correction to the already-closed window, or put it in a side output for separate handling. Flink lets you configure all three via the allowedLateness setting.

This is the concept that separates engineers who have shipped real ML systems from those who have only trained models. The failure is so common it has a name: point-in-time leakage (also called future leakage or look-ahead bias).

The failure, in plain words: when you build a training dataset, you join features to labels. If you join on entity_id without considering timestamps, you may attach a feature value that was computed after the label event occurred. The model trains on data that would have been impossible to know at prediction time. Offline metrics look great; production metrics are terrible. The model learned to cheat.

Concrete scenario: you are building a churn model. A user either cancels (label = 1) or does not (label = 0) in a given month. You want to use their "total_purchases_last_30_days" as a feature. The label event (cancellation) occurs on Day 15 of October. If you compute total_purchases_last_30_days using data from October 1–31, you have included purchases that happened after the user cancelled. The model learns that "users who made purchases after cancelling don't churn" — which is nonsense, but the signal is so strong that offline AUC looks 0.05 better than reality.

The 4-row table showing the wrong join and the right join:

The correct approach is a point-in-time join: for each training example, compute every feature value as it existed at the moment of the label event. In SQL this is an AS OF join (supported by Feast, Tecton, and time-travel queries in Iceberg/Delta). In practice: your offline feature store must store the full history of each feature, keyed by (entity_id, timestamp), and the join filters to feature_timestamp <= label_timestamp with the latest value that satisfies the constraint.

Suppose your team defines a feature: user_avg_purchase_value_30d — the mean value of purchases by a user in the last 30 days. Sounds simple. Here is what actually happens without a feature store:

Training (Python/Spark, data scientist's laptop or cluster):

import pandas as pd

def user_avg_purchase_30d(user_id, as_of_date, purchases_df):
    window = purchases_df[
        (purchases_df["user_id"] == user_id) &
        (purchases_df["date"] >= as_of_date - pd.Timedelta(days=30)) &
        (purchases_df["date"] < as_of_date)
    ]
    return window["value"].mean()  # NaN if no purchases

Serving (Java microservice, written by a backend engineer 3 months later):

// Java equivalent — seemingly identical
double userAvgPurchase30d(String userId, Instant asOf, List<Purchase> purchases) {
    OptionalDouble avg = purchases.stream()
        .filter(p -> p.userId.equals(userId))
        .filter(p -> p.date.isAfter(asOf.minus(30, DAYS))
                  && p.date.isBefore(asOf))
        .mapToDouble(p -> p.value)
        .average();
    return avg.isPresent() ? avg.getAsDouble() : 0.0;  // <-- BUG: 0.0 not NaN
}

Spot the difference: Python's mean() returns NaN for users with no purchases; Java's fallback returns 0.0. Every new user gets feature value 0.0 in production and NaN during training (which the model imputed differently). The model was trained on imputed values; it now gets a different distribution in production. Worse: this is silent. There's no error. The model just performs worse for new users, and you find out months later via A/B test.

This is training-serving skew. It is so common that it has its own name. A feature store fixes it by ensuring one canonical definition runs on one codebase — the feature store SDK calls the same underlying logic for both offline backfill and online serving.

A feature store is not one system — it is three systems with different storage technologies and access patterns, connected by a shared feature definition registry.

The key operational challenge is keeping offline and online in sync: the offline store and online store must be populated by the same feature transformation logic, even though they run at different frequencies and on different infrastructure. Feature store frameworks (Feast, Tecton) solve this by generating both the batch backfill job and the streaming materialization job from the same feature definition.

Not all features need the same freshness. Requiring all features to be real-time is expensive and operationally complex; allowing all features to be day-stale hurts model quality for anything time-sensitive. The solution is tiered freshness: choose the staleness each feature can tolerate, and pick the cheapest infrastructure that meets that requirement.

The cost cliff: moving from daily to near-real-time typically costs 5–10× more compute. Moving from near-real-time to sub-10-second streaming may cost another 3–5×. The right engineering question is always: what is the marginal model quality gain from fresher data, divided by the marginal infrastructure cost? For many features, daily is fine — users don't change their 90-day purchase history in the last hour.

An embedding (a user vector from the retrieval tower, an item vector from a content encoder) is just a feature with extra logistics. Two serving patterns:

The version problem is sharper for embeddings than scalars: a vector from encoder v7 is meaningless in the geometry of encoder v8. Store the encoder version with every vector, and never mix versions between a query tower and an ANN index built from a different version — this is the classic "retrieval quality silently collapsed" incident.

• Every team re-implements features. Five teams compute "user 7-day CTR" five ways (different windows, different null handling). Models disagree for reasons nobody can explain.
• Training-serving skew by construction. Training reads a SQL pipeline; serving reads a Java service. Two codebases drift; offline metrics stop predicting online behavior.
• Leakage everywhere. Without point-in-time APIs, every team writes its own joins, and someone always joins tomorrow's aggregate onto today's label.
• No lineage. A bad upstream table corrupts twenty models and nobody can enumerate which ones — the registry IS the blast-radius map.

A modern CPU is optimized for latency: branch prediction, out-of-order execution, huge caches — all designed to finish one instruction chain as fast as possible. A GPU is optimized for throughput: thousands of simple ALU cores that execute the same instruction on thousands of data elements simultaneously (SIMT — Single Instruction, Multiple Threads).

The key operation in neural networks is the matrix multiply: for a layer with weight matrix W ∈ ℝ^{m×k} and input batch X ∈ ℝ^{k×n}, computing Y = WX requires 2·m·k·n floating-point operations — and every single one is independent of every other one. That is the exact shape of work a GPU loves.

The 1000× arithmetic gap is why you cannot train a large model on CPU in any reasonable time.

Understanding the hierarchy is the key to understanding every performance bottleneck:

The rule: data flows up the hierarchy to be computed on, and back down when done. The bottleneck is almost always the slowest link that data must cross.

Arithmetic intensity is the ratio of FLOPs performed to bytes of data moved from HBM. It determines whether a kernel is limited by compute or by memory bandwidth.

An H100 has a ridge point (also called the roofline crossover) of roughly 2000 TFLOP/s ÷ 3.35 TB/s ≈ 600 FLOP/byte. A kernel with intensity above 600 is compute-bound; below 600 it is memory-bandwidth-bound.

One training step has five phases. Understanding each phase lets you attribute time and memory correctly:

1. Load batch: CPU fetches a mini-batch from the dataset (often async/prefetch in a DataLoader worker), pins it in CPU RAM, and DMA-transfers it to GPU HBM via PCIe. Cost: PCIe bandwidth × batch size × token size.
2. Forward pass: Execute each layer in order. For a transformer: embedding lookup → attention → FFN → … → logits. Each layer reads weights from HBM, computes (matmuls, activations), and writes activations back to HBM (they'll be needed in the backward pass).
3. Loss computation: Compare logits to labels; compute cross-entropy (or task loss). This produces one scalar loss value and a gradient ∂L/∂logits.
4. Backward pass: Backpropagation. For each layer (in reverse), compute gradients of the loss w.r.t. that layer's parameters. This reads the saved activations from the forward pass — which is why activations live in memory the entire forward+backward cycle.
5. Optimizer step: Use gradients to update parameters. Adam needs optimizer state (first and second moment) per parameter. This is the most memory-hungry single operation.

This is the most important memory calculation in training system design. Let's derive it from scratch.

A 7B-parameter model (think LLaMA 2 7B) has 7 × 10⁹ parameters. Let P = 7 × 10⁹.

The problem with fp32-only training: large models × 4 bytes/param × optimizer state = enormous memory. Moving to fp16 halves most allocations.

The problem with fp16-only training: fp16 has a narrow dynamic range (max value ~65504). Gradient values often underflow (become zero) or overflow (explode) during backprop, causing training instability or divergence.

The mixed-precision solution (Micikevicius et al., 2018):

1. Forward and backward passes in bf16 (or fp16) — fast tensor-core matmuls, low HBM bandwidth
2. Loss scaling: multiply the loss by a large scalar (e.g., 2¹⁵) before backward, divide gradients afterward — prevents fp16 underflow
3. Maintain a full fp32 "master copy" of weights and optimizer state — prevents accumulated rounding errors in the weight update
4. Copy fp32 master → bf16 working copy before each forward pass

bf16 vs fp16: bf16 has the same exponent bits as fp32 (8 bits), giving the same dynamic range. fp16 has only 5 exponent bits. For LLM training, bf16 is strongly preferred because gradient values span a wide magnitude range and bf16 rarely needs loss scaling. Modern TPUs and H100s support bf16 natively.

The activation memory problem: during backprop, gradients for layer k require the activations from layer k's forward pass. With 32 layers and large hidden dimensions, activations easily dwarf even the 112 GB static memory. For LLaMA 7B at batch=32, seq=2048: 32 sequences × 2048 tokens × 4096 dim × 2 bytes × 32 layers ≈ 16 GB just in activations.

Gradient checkpointing (activation checkpointing): instead of saving every layer's activations during the forward pass, save only at checkpointed layers (e.g., every 4th layer). During the backward pass, when gradients for a non-checkpointed layer are needed, recompute that layer's forward pass from the most recent checkpoint.

Suppose training a well-sized language model takes six months on a single GPU. You have four GPUs. The most natural idea: each GPU trains on a different quarter of the data, and you combine their gradients. This is data parallelism.

The key insight: because every GPU holds a full copy of the model and sees different data, all replicas produce independent gradients for the same parameter. Averaging those gradients — which is mathematically equivalent to training on all the data combined — is called allreduce.

Let's trace one step concretely. Model has parameters W (just one matrix for simplicity), and we have 2 GPUs.

1. Replicate: Both GPU-0 and GPU-1 hold a full copy of W.
2. Partition batch: Global batch of 64 samples → GPU-0 gets samples 0–31, GPU-1 gets 32–63.
3. Forward + backward (parallel): Each GPU computes forward and backward on its half of the batch, producing gradients g₀ (on GPU-0) and g₁ (on GPU-1).
4. Allreduce: GPUs communicate to compute g_avg = (g₀ + g₁) / 2. After allreduce, both GPUs hold the same g_avg.
5. Optimizer step: Each GPU applies the same optimizer step to its local copy: W ← W - lr × optimizer(g_avg). Both copies stay identical.
6. Next step: Repeat. Replicas stay synchronized indefinitely.

This is DDP (Distributed Data Parallel) in PyTorch. In practice, PyTorch overlaps allreduce with the backward pass: as soon as a parameter's gradient is computed it is immediately allreduced while backprop continues on earlier layers — this hides most communication latency behind compute.

Allreduce must sum a tensor across N GPUs and give every GPU the result, using as little bandwidth as possible.

Naive approach: send all gradients to a parameter server, sum them, broadcast back. Communication volume = 2 × gradient_size per GPU. The parameter server is a bottleneck.

Ring allreduce: arrange N GPUs in a logical ring. Run two phases:

1. Scatter-reduce: Each GPU sends a chunk of its gradient to the next GPU in the ring and receives a chunk from the previous GPU; accumulates. After N-1 steps, each GPU holds the fully-reduced sum for one chunk of the gradient tensor.
2. Allgather: Each GPU broadcasts its fully-reduced chunk around the ring. After N-1 more steps, every GPU holds the complete reduced gradient.

For large models (gradient tensor = tens of GB), even optimal allreduce takes seconds per step on slow interconnects — which is why DDP requires high-bandwidth links (NVLink within a node, InfiniBand across nodes) to remain efficient.

In naive DDP, every GPU holds an identical copy of:

• Model parameters (fp32 master + bf16 working = 6P bytes)
• Gradients (2P bytes)
• Optimizer state: Adam m + v = 8P bytes

Total: 16P bytes per GPU. Every byte is redundant — GPU-1 holds exactly the same optimizer state as GPU-0, for the same parameters. This is memory wasted purely for the convenience of not communicating.

At N=64 GPUs, you are storing 64 full copies of the optimizer state. For a 70B model, optimizer state alone is 70B × 8 bytes = 560 GB, replicated 64 times = 35.8 TB of aggregate HBM. Replicated to identical garbage.

ZeRO asks: what if we shard the redundant parts?

ZeRO (Zero Redundancy Optimizer, Rajbhandari et al. 2020) has three stages, each eliminating more redundancy:

At N=64 GPUs and a 7B model (P = 7B): ZeRO-3 per-GPU memory ≈ 12 × 7B / 64 bytes = 12 × 109 MB ≈ 1.3 GB static per GPU — plus activations. The 80 GB HBM per H100 becomes almost entirely available for activations and larger batches.

Why gradient accumulation exists: large effective batch sizes improve training stability and convergence (up to a point), but large batches require more GPU memory for activations. If a batch of 512 sequences doesn't fit, you can process 8 micro-batches of 64 sequences and accumulate gradients before calling the optimizer step. This is gradient accumulation.

optimizer.zero_grad()
for micro_batch in split_batch(batch, n_accumulation_steps=8):
    loss = model(micro_batch) / 8   # scale loss
    loss.backward()                 # accumulates grads
optimizer.step()                    # one update per 8 micro-batches

This is mathematically equivalent to training on a batch 8× larger — at the cost of 8× more compute per optimizer step (no parallelism benefit from the accumulation).

The global batch size / LR scaling caveat: when you scale from 1 GPU (batch B) to N GPUs (effective batch N×B), you are changing the statistical properties of each gradient update. The linear scaling rule (Goyal et al., 2017) says: multiply the learning rate by N when you multiply the batch size by N. This works up to a point (typically up to 8K-32K tokens per batch for LLMs); above this, you often need learning rate warmup and the rule breaks down, requiring hyperparameter re-tuning.

Before reaching for parallelism strategies, let's quantify the problem precisely. Consider a 70 B-parameter model (similar to LLaMA-2 70B).

In bf16: 70 × 10⁹ × 2 bytes = 140 GB — already 1.75× the 80 GB capacity of an H100 SXM5, and we haven't stored a single gradient or optimizer state yet.

Even at inference (no gradients, no optimizer states), 140 GB weights alone won't fit one H100. Model parallelism is not an optimization — it is a prerequisite.

The fundamental operation in a transformer is the matrix multiply: Y = X W. Tensor parallelism (TP) splits this multiply across GPUs so each holds only a slice of W and does a proportional slice of the compute.

Worked example: 4×4 matmul on 2 GPUs

Say X is a 4×4 input and W is a 4×4 weight matrix. We want to compute Y = X W.

Column-parallel (forward pass — split W vertically):

W = [ W_A | W_B ]        # W_A on GPU-0 (4x2), W_B on GPU-1 (4x2)

GPU-0: Y_A = X @ W_A    # shape (4,2)
GPU-1: Y_B = X @ W_B    # shape (4,2)

Y = concat(Y_A, Y_B, dim=1)   # shape (4,4) — no communication yet

Each GPU receives the full input X (needs a broadcast at the start of the layer) and computes half the output columns independently. The results are concatenated, not summed.

Row-parallel (backward / second linear layer — split W horizontally):

W = [ W_C ]   # rows 0-1 on GPU-0 (2x4)
    [ W_D ]   # rows 2-3 on GPU-1 (2x4)

GPU-0 gets X_top (4,2): partial_0 = X_top @ W_C   # shape (4,4)
GPU-1 gets X_bot (4,2): partial_1 = X_bot @ W_D   # shape (4,4)

Y = allreduce(partial_0 + partial_1)              # sum partials, then sync

Here the input is already split (from the column-parallel step), and the outputs are partial sums that must be reduced across GPUs with an allreduce.

Tensor parallelism splits individual operations. Pipeline parallelism (PP) takes a different approach: split the layers of the model across GPUs, so GPU-0 holds layers 1–N/P, GPU-1 holds layers N/P+1 through 2N/P, and so on. Data flows through GPUs like an assembly line.

The bubble problem

With a single microbatch and P pipeline stages, naive execution looks like this (F = forward pass through one stage, B = backward):

Stage 0:  [F0][ ][ ][ ][B0]
Stage 1:       [F1][ ][ ][ ][B1]
Stage 2:            [F2][ ][ ][ ][B2]
Stage 3:                 [F3][ ][ ][ ][B3]
           ^ bubble fills P-1 stages idle ^

The "bubble" — idle time while stages wait for activations from upstream — wastes a fraction of compute. With P stages and M microbatches:

Concrete example: P = 4 stages, M = 1 microbatch → bubble = 3/4 = 75% waste. P = 4, M = 16 → bubble = 3/19 ≈ 16% waste. Target M ≥ 4×P for the bubble to fall below ~20%.

1F1B schedule (interleaved)

The 1F1B (one-forward-one-backward) schedule overlaps forward and backward passes for different microbatches, dramatically cutting the bubble:

Stage 0:  [F0][F1][F2][F3][B3][B2][B1][B0]
Stage 1:       [F0][F1][F2][B2][B1][B0]
Stage 2:            [F0][F1][B1][B0]
Stage 3:                 [F0][B0]

Megatron-LM's interleaved pipeline further reduces the bubble by assigning non-contiguous "chunks" of layers to each stage (e.g., stage 0 owns layers 1–4 and 33–36 of a 64-layer model). This keeps every GPU busy at the cost of more inter-stage communication.

Both TP and PP split a single dense model. Mixture-of-Experts (MoE) takes a fundamentally different approach: instead of one large feedforward network (FFN), replace it with N expert FFNs, and route each token to only the top-k of them.

How MoE works

In a standard transformer FFN layer, every token uses the same weights. In an MoE layer:

1. A lightweight router (a small linear layer + softmax) scores each token against all N experts.
2. Only the top-k scores are retained (typically k=1 or k=2).
3. The token is sent to those k experts; their outputs are summed (weighted by the router score).

# Simplified MoE forward pass
router_logits = token_hidden @ W_router        # shape: (seq_len, num_experts)
scores = softmax(router_logits, dim=-1)
top_k_indices = argsort(scores, descending=True)[:, :K]

output = zeros_like(token_hidden)
for expert_id in unique(top_k_indices):
    mask = (top_k_indices == expert_id).any(dim=1)
    expert_input = token_hidden[mask]
    expert_output = experts[expert_id](expert_input)   # full FFN
    output[mask] += scores[mask, expert_id] * expert_output

Why this matters: A 100B-parameter MoE model with 8 experts and k=2 activates only ~25B parameters per token. You get 100B of learned capacity (better quality) at the inference cost of a ~25B dense model. Mixtral, GPT-4, and Gemini 1.5 use this architecture.

Expert parallelism

With expert parallel (EP), each GPU holds a subset of experts. Tokens are routed to the correct GPU via all-to-all collective: GPU-0 sends the tokens destined for expert-4 to GPU-1, receives tokens for its own experts, runs them, then all-to-alls again to return results.

For very long sequences (32k–1M tokens), activations dwarf parameters in memory. Sequence parallelism (SP) partitions the sequence dimension across the same TP ranks used for weight splitting:

• Attention: each rank handles a contiguous chunk of query/key/value positions; Ring Attention overlaps the KV-tile fetches with compute to avoid blocking.
• FFN layers: column/row parallel already; sequence parallelism adds LayerNorm and dropout over non-TP dimension.
• Communication: allgather at the start of each TP operation, reduce-scatter at the end (replacing the full allreduce — same bandwidth but lower peak memory).

Mean time between failures (MTBF) for a single modern GPU or server is roughly 3 years (≈ 26 280 hours). That sounds reassuring until you multiply by cluster size.

Plug in numbers for a 10 000-GPU run:

That single number explains why every serious training framework has checkpointing wired in by default: without it, a 10k-GPU job loses its entire state every few hours.

Before deciding checkpoint frequency, you need to know how expensive each checkpoint is. Work through the byte math for a 7B-parameter model trained with Adam in mixed precision:

Writing 126 GB to a high-performance parallel filesystem (say, 10 GB/s sustained) takes about 12–13 seconds. On a slower NFS or object store (1 GB/s), that's 2 minutes. During a synchronous checkpoint every GPU stalls — you lose 2 minutes of training throughput every checkpoint interval.

Choosing checkpoint frequency: balance the cost of a checkpoint against the expected lost work. If the cluster MTBF is 2.6 h and each checkpoint costs 0.5% throughput, checkpointing every 10 minutes loses about 5% throughput but caps replay to 10 minutes of compute. Checkpointing every hour saves throughput but risks losing 60 minutes on failure. The rule of thumb at large scale: checkpoint every 10–20 minutes, async.

Checkpointing handles crash recovery. Several other disciplines prevent slower, harder-to-diagnose pathologies.

The loss curve is the heartbeat monitor of your training job. Every shape has a cause. This table is a beloved interview probe — interviewers will describe a curve and ask what you do.

At small scale, a researcher manually tries learning rates. At large scale, that approach wastes millions of dollars of GPU time. Two practices separate serious infrastructure from hobby setups.

Experiment tracking: every run must log, automatically: hyperparameters (LR, batch size, warmup steps, model config), system metrics (GPU utilization, throughput tokens/sec, memory), and training metrics (loss per step, gradient norm, loss-scale events). Tools: MLflow, Weights & Biases, TensorBoard. The non-negotiable invariant: given a run ID, you can reproduce that run exactly — seeds, data version, code commit, config.

Hyperparameter search: naive grid search is exponential. The modern default is ASHA (Asynchronous Successive Halving Algorithm): launch many trials with different configs; after a short horizon (say, 1000 steps), kill the bottom half by validation loss; double the budget for survivors; repeat. This gives near-optimal results with a fraction of the compute of full-budget grid search. Key insight: trial ranking stabilizes early — a run that is bottom-quartile at 1k steps is almost always bottom-quartile at 100k steps, so killing it early is low-risk and high-reward.

# Minimal checkpoint manifest
{
  "step": 47200,
  "model_state_dict": "...",       # sharded across ranks in ZeRO-3
  "optimizer_state_dict": "...",   # Adam m, v, step count
  "lr_scheduler_state_dict": "...",# cosine schedule offset
  "rng_state": {                   # per-rank
      "python": "...",
      "numpy": "...",
      "torch_cpu": "...",
      "torch_cuda": "..."
  },
  "dataloader_state": {
      "epoch": 2,
      "shard_index": 14,
      "offset_within_shard": 8192
  },
  "config_hash": "a3f2...",        # git commit + config hash for reproducibility
  "experiment_id": "run_0042"
}

In practice with ZeRO-3 or FSDP, model_state_dict and optimizer_state_dict are sharded: each rank saves only its slice, and a manifest file records how to reassemble them. PyTorch's torch.distributed.checkpoint handles this natively.

Gradient norm is one of the most useful diagnostic signals and is almost free to log. The global gradient norm is computed before clipping:

Healthy training: the norm hovers in a stable range (often 0.5–3 for language models with clip threshold 1.0). Interpret deviations:

Here is what a production training run looks like with all disciplines in place:

1. Pre-run checklist: seed set, experiment ID registered in tracking system, config hash recorded, data version pinned, checkpoint destination with write-access verified.
2. First 1 000 steps: watch loss curve, gradient norms, and GPU utilization closely. A bug in data pipeline or init usually shows up here. Do a manual checkpoint at step 500 to verify the checkpoint roundtrip works.
3. Steady state: async checkpoint every 10–15 minutes; keep last 3 checkpoints (ring buffer); alert on cluster-health events; log per-rank throughput to catch stragglers.
4. On failure: identify the failed rank(s), evict from job, optionally elastic-replace; resume from last checkpoint; verify loss picks up at the right level (not a spike).
5. On loss anomaly: follow the 6-step triage. Do not panic-restart. Document the event in the experiment log with step number, gradient norm at the spike, and resolution.
6. End of job: convert checkpoint to serving format; run validation eval; register in model registry with lineage (data version, config, commit, training metrics).

Every serving system optimizes a three-way tension:

These three are interlinked: lowering latency often cuts throughput (fewer requests in flight at once), and raising throughput often raises per-request latency (requests queue longer). Cost tracks with how well you keep the GPU busy — an idle GPU is maximum cost-per-request.

The single most important lever is batch size: squeezing more requests through the same forward pass amortizes the fixed overhead of loading weights, launching kernels, and moving data. We will quantify this shortly.

Engineers often optimize for p50 (median) latency, but production pages call many services in parallel. The user sees the maximum — and probability theory makes that brutal.

Concrete example. Suppose one service call has p99 latency = 50 ms, meaning each call is fast 99% of the time. A page that fans out to 50 independent services is fast only when ALL 50 are fast:

So even though each individual service is fast 99% of the time, nearly 40% of page loads are slow. Push that to 100 services and you get only 37%. This is why Staff+ engineers quote p99 and p999 — not p50 — and why tail-latency SLOs cascade through every dependency.

Practical takeaway: a 10 ms p50 improvement is worth less than a 5 ms p99 improvement when your service is in a fan-out call graph.

A GPU is a throughput machine. It executes thousands of threads in parallel, but loading weight tensors into SRAM, launching CUDA kernels, and synchronizing results all have fixed overhead that is paid once per batch, not once per request.

Concrete numbers. Suppose a model takes 10 ms for a single forward pass. Because of the fixed overhead:

• Batch size 1 → 10 ms → 100 requests/sec
• Batch size 4 → 11 ms → 364 requests/sec (3.6× throughput for 10% extra latency)
• Batch size 32 → 12 ms → 2,667 requests/sec (26.7× throughput for 20% extra latency)
• Batch size 128 → 20 ms → 6,400 requests/sec (64× throughput for 2× latency)

The shape: throughput scales nearly linearly with batch size until the GPU is compute-saturated; beyond that adding more requests mostly adds latency without adding throughput. The sweet spot — and finding it for your model/hardware pair — is a core serving-engineer task.

In production, requests do not arrive in neat synchronized groups. Dynamic batching solves this: the server accumulates incoming requests in a queue and fires the GPU once either (a) the batch is full or (b) a maximum wait time elapses.

Two parameters control the tradeoff:

Example. With max_batch=32 and max_wait=5 ms, under high load the batch fills before 5 ms and latency stays low. Under low load the batch fires after 5 ms with only a few requests — the latency floor is now 5 ms worse. This 5 ms is your queuing tax, and it must fit within your SLO budget.

Little's law is the single most useful queueing result for capacity planning. In plain words: the average number of requests in a system equals the average arrival rate times the average time a request spends in the system.

Worked example. Your model server handles 200 requests/sec (λ = 200). Each request takes on average 80 ms end-to-end (W = 0.08 s). Then:

Why it matters for capacity planning. If you want to handle λ = 500 req/s at W = 80 ms, you need L = 40 concurrent slots. If each GPU thread pool handles 8, you need 5 GPU workers. The law lets you convert a QPS target into a hardware count with just two numbers.

Rearranged for latency budgeting: W = L / λ. If you cap concurrency at L = 20 (e.g., memory limit), and traffic is λ = 300 req/s, then average wait time W = 20/300 ≈ 67 ms — that is your achievable average latency floor.

Every model server — whether Triton, TorchServe, or a custom gRPC service — follows the same skeleton:

1. Request queue. Incoming requests are enqueued. This decouples the network-facing threads from the GPU-facing threads, absorbs bursts, and is where backpressure is applied (reject or shed load when the queue is full).
2. Batcher. Pulls from the queue, forms a batch (up to max_batch_size or max_wait_ms — see ch9), and passes it to the runtime.
3. Runtime. Executes the model on the batch. This is TensorRT, ONNX Runtime, PyTorch, TorchScript, or a compiled kernel. It owns the GPU context and the loaded weights.
4. Response path. Splits the batch result back into per-request responses, attaches metadata (latency, model version), and sends replies.

Around this skeleton you add: pre/post-processing steps (tokenization, embedding lookup, output decoding), health-check endpoints, Prometheus metrics, and a sidecar for logging.

Not every model needs a GPU. The decision hinges on three numbers: model size (how many FLOPs/bytes per inference), QPS (how many inferences per second), and latency budget (how many ms you have).

Worked example. A recommendation light ranker has 20M parameters, needs to score 500 candidates in 10 ms at 50 req/s: 50 × 500 = 25,000 scoring calls/s but each is tiny. A multi-core CPU cluster (32 cores) at ~2ms/batch can handle this. A heavy ranker with 200M params and p99 < 5 ms at 2,000 req/s needs GPU batching.

For a stateless web service, horizontal autoscaling is easy: spin up a new instance in 2–5 seconds, it's ready to serve. For a GPU model server, the cold-start problem changes the calculus:

1. Model load time. A 7B parameter model in fp16 = 14 GB. Loading from S3/GCS to GPU HBM at ~2 GB/s takes 7 seconds minimum. In practice, with storage overhead and driver init, cold start is 30–120 seconds for large models.
2. GPU driver + CUDA context init. Even a blank GPU context takes 2–10 seconds to initialize.
3. Compilation step. TensorRT compilation (engine build) can take minutes for a new model. This is usually done offline and cached, but any cache miss is fatal for a cold start.

This means you cannot rely on autoscaling to absorb fast traffic spikes the way you can with stateless services. A 90-second cold start is 90 seconds of degraded capacity during which queues build and SLOs are breached.

Solutions:

Not all requests need a fresh model inference. Three caching layers reduce load and cost:

Hit-rate math example. Suppose 1M distinct queries per day, but the top 10,000 queries account for 60% of traffic (power-law distribution, common in search/recommendation). A result cache of 10,000 entries achieves a 60% hit rate. At 10,000 req/s total, 6,000 req/s hit the cache (1 ms Redis lookup) and 4,000 req/s go to the GPU (15 ms inference). Effective average latency: 0.6 × 1 + 0.4 × 15 = 6.6 ms. Without cache: 15 ms. And you need 10,000 × 15 ms = 150,000 ms of GPU work per second vs. 4,000 × 15 ms = 60,000 ms — a 2.5× GPU cost reduction.

A serving system that returns an error under load is a bad product. The gold standard is graceful degradation: as the system comes under stress, it falls back to progressively simpler (but still useful) responses rather than failing entirely.

The degradation ladder. Design your system with at least three rungs:

1. Full model. Normal path. Latest model, full feature set. SLO: 15 ms p99.
2. Smaller model / cached embedding. A lighter version (distilled, quantized, or an older version that's still loaded). Same API, worse quality. SLO: 8 ms p99. Activate when GPU load > 90% or p99 > 25 ms.
3. Popularity / editorial baseline. Return pre-computed "top-N most popular items" or a static default. No ML inference needed. SLO: 1 ms. Activate when the model fleet is entirely down.

Circuit breakers implement the switch automatically. A circuit breaker wraps calls to the model server. It tracks error rate or latency over a rolling window. When the error rate exceeds a threshold (e.g., 5% over 30 seconds), the circuit "opens" and all calls immediately go to the fallback path without even attempting the model — preventing cascading timeouts. After a "half-open" probe period, it retries the primary and closes if healthy.

A production system often has dozens of models: ranking model, CTR predictor, safety classifier, embedding model, re-ranker. Running each on a dedicated GPU is expensive and leads to low average utilization (each model might peak at different times of day).

GPU bin-packing co-locates multiple models on the same GPU, multiplexing the compute. Considerations:

• Memory partitioning. Each model needs a memory slice. NVIDIA MIG (Multi-Instance GPU) hardware-partitions an H100 into up to 7 isolated GPU instances, each with their own HBM slice — no model can interfere with another's memory.
• Compute multiplexing. Without MIG, CUDA streams allow multiple models to share compute with some contention. Good for light models; risky for latency-sensitive large models.
• Load isolation. A traffic spike on one model should not spike latency for co-located models. MIG provides hard isolation; CUDA streams do not.

A/B testing at the serving layer. Rather than a separate A/B infrastructure, many teams implement model experiments directly in the serving tier: the serving process reads a config that maps a fraction of traffic to "model A" and the rest to "model B". Requests are assigned deterministically by user-ID hash. This removes the need for separate deployments and lets experiments ramp/ramp-down instantly via config changes.

Every optimization has a cost — engineering time, accuracy risk, infrastructure complexity. Work cheapest-first. The canonical ordering:

The rule: exhaust steps 1–3 before touching the training pipeline. Steps 4–6 are only justified when you have an accuracy budget to spend and a large enough fleet that the engineering cost amortizes.

A 32-bit float can represent roughly 4 billion distinct values. For most trained weights, this is overkill — the weight distribution is narrow and roughly Gaussian. Quantization maps this float range into a much smaller set of integers, using a scale and zero-point so you can reconstruct an approximate float later.

Worked example — int8 quantization of a tiny weight tensor. Suppose our layer has four weights:

W = [0.8,  -0.4,  1.2,  -1.0]   # fp32, 4 × 4 bytes = 16 bytes

Step 1: find the range. w_min = -1.0, w_max = 1.2.

Step 2: compute scale and zero-point for int8 (range [-128, 127]):

Step 3: quantize each weight: q = round(w / s) + z

q(0.8)  = round(0.8  / 0.00863) + (-12) = round(92.7)  - 12 =  81
q(-0.4) = round(-0.4 / 0.00863) + (-12) = round(-46.3) - 12 = -58
q(1.2)  = round(1.2  / 0.00863) + (-12) = round(139.0) - 12 = 127
q(-1.0) = round(-1.0 / 0.00863) + (-12) = round(-115.9)- 12 = -128
W_int8 = [81, -58, 127, -128]   # int8, 4 × 1 byte = 4 bytes  (4× smaller)

Step 4: dequantize (at multiply time): w_approx = (q − z) × s

w_approx(81)   = (81  - (-12)) × 0.00863 = 93  × 0.00863 ≈  0.803  (true: 0.8)
w_approx(-128) = (-128-(-12)) × 0.00863 = -116 × 0.00863 ≈ -1.001  (true: -1.0)

The error is small (≈1%). The memory footprint dropped 4×. For a matmul, the GPU can now move 4× more weights per second through HBM — and memory bandwidth is usually the bottleneck in decode.

What breaks in LLMs — outlier channels. LLMs (unlike CNNs) develop a small number of activation channels with massive magnitudes — sometimes 100× larger than the rest. Per-tensor int8 quantization clips these outliers catastrophically. Solutions:

Training a student model on labels alone wastes signal. The teacher's full output distribution — its soft probabilities over all classes — contains rich information: it knows that a cat looks a little like a dog, that "Paris" is probably followed by "is" or "the" rather than random tokens. Distillation feeds this signal to the student.

The setup. Teacher model T (large, expensive) and student model S (small, cheap). For each training example x:

1. Run T(x) and collect the soft logits (or probabilities at temperature τ > 1 to soften the distribution).
2. Train S to minimize a mix of: (a) cross-entropy with hard labels, (b) KL divergence from teacher's soft distribution.

When distillation beats direct training. If you train a small model from scratch on hard labels alone, it often underperforms a distilled student of the same size — especially when labels are sparse or the task is complex. The teacher provides dense, calibrated supervision on every example. Rule of thumb: distill when the student is ≤ ¼ the teacher's parameter count and you have the teacher already trained.

The hidden cost: kernel launch overhead and memory round-trips. A modern GPU operation (GELU, LayerNorm, softmax) is not one operation — it's a sequence of separate CUDA kernels, each with launch overhead (~5–10µs) and each reading and writing its result back to HBM (GPU main memory). For small tensors or fast ops, these round-trips dominate the actual computation.

Fusion: the fix. A fused kernel computes multiple ops in a single pass, keeping intermediate results in fast SRAM (on-chip registers/shared memory) rather than writing them to HBM between steps.

Worked example: 3 elementwise ops → 1 kernel.

# Unfused (3 separate kernels, 3 HBM round-trips)
x = dropout(x)        # kernel 1: read x, write x'  → HBM
x = x + residual      # kernel 2: read x', residual; write x'' → HBM
x = layer_norm(x)     # kernel 3: read x''; write x_out → HBM

# Fused (1 kernel, 1 HBM round-trip)
x_out = fused_dropout_add_layernorm(x, residual)
# Intermediate values stay in registers/shared mem; only x_out hits HBM

For a tensor of shape (batch=32, seq=512, dim=4096) at fp16 (2 bytes/elem), each HBM round-trip moves 32 × 512 × 4096 × 2 ≈ 134 MB. Three round-trips = 402 MB; fused = 134 MB. With H100 HBM bandwidth of ~3 TB/s, that saves ~90µs per layer. Multiply by 32 layers and 100 tokens/request: real latency gains.

How to get fusion in practice:

You run a 7B-parameter chat model on A100-80GB GPUs. Current setup: fp16 weights, no compilation, batch size 8, throughput ~400 tokens/sec/GPU, cost ~\$0.003 / 1k tokens. Target: \$0.00075 / 1k tokens (4× reduction). Here is how you walk the lever ladder:

Baseline cost arithmetic:

GPU cost:     ~\$2/hr = \$0.000556/sec
Throughput:   400 tokens/sec
Cost/token:   0.000556 / 400 = \$0.00000139 = \$0.00139/1k tokens

Wait — that's already < \$0.003. Let's be more realistic:
Effective throughput at batch=8 with p99 latency SLO:  ~200 tokens/sec
Cost/1k tokens:  0.000556 / 0.2 = \$0.00278/1k tokens  ✓ matches stated baseline

Lever 1 — torch.compile + FlashAttention (free): Adds ~25% throughput. New: ~250 tok/s. Cost: ~\$0.00222/1k. Gain: 1.25×. No accuracy risk. Ship it first.

Lever 2 — int8 weight quantization (PTQ): 7B model weights: 7 × 10⁹ × 2 bytes (fp16) = 14 GB → int8 = 7 GB. Now two model replicas fit on one 80GB GPU (previously one replica + KV cache). With better batching enabled by dual-replica, effective throughput: ~450 tok/s. Cost: ~\$0.00124/1k. Cumulative gain vs baseline: 2.24×. Accuracy: run eval suite; typical degradation < 0.5% on standard benchmarks.

Lever 3 — increase batch size / continuous batching: With int8 freeing memory, bump effective batch to 32. Throughput: ~600 tok/s (GPU now more compute-bound). Cost: ~\$0.00093/1k. Cumulative gain: 2.99×. Still on the same GPU count.

Lever 4 — int4 quantization (GPTQ/AWQ, calibrated): 7B × 0.5 bytes = 3.5 GB weights. Four replicas per GPU possible. Throughput jumps to ~1000 tok/s with batching. Cost: ~\$0.00056/1k. Cumulative gain: 4.96× — exceeds the 4× target. Accuracy: GPTQ at 4-bit shows ~1–2% MMLU degradation; needs eval gate before production.

Decision: Ship compilation + int8 immediately (gain 2.24×, near-zero risk). Schedule int4 + GPTQ behind an accuracy eval gate. Do not distill — you don't need to, and distillation would take 3–6 weeks of training.

The right optimization depends on which resource is the bottleneck. Two questions to ask first:

Concretely: an H100 has ~3 TB/s HBM bandwidth and ~2000 TFLOP/s bf16. For a 7B int8 model weight matrix (7 GB), bandwidth-limited decode throughput ≈ 3 TB/s ÷ 7 GB ≈ 430 tokens/sec theoretical maximum per GPU — before batching tricks. This is why memory bandwidth is the first constraint to reason about in LLM serving.

Every mature ML team runs candidates through a fixed sequence of checkpoints before a model touches all users. Skipping a rung is tempting — it saves days — and is almost always regretted. Here is the standard five-rung ladder.

1. Offline evaluation — held-out dataset metrics (AUC, NDCG, BLEU, accuracy…)
2. Shadow traffic — real requests, production twin, no user impact
3. Canary (1–5%) — live traffic slice, full product integration, small blast radius
4. A/B test (10–50%) — controlled experiment, statistical inference on goal and guardrail metrics
5. 100% rollout + holdback — full traffic, small control held back for continued monitoring

The whole point of the ladder is that each rung exposes failure modes that earlier rungs are structurally blind to. Memorize this table.

Shadow mode (also called "dark launch" or "shadow serving") is the single most underrated tool in the deployment toolbox. The idea: every incoming production request is duplicated. The original goes to the live model as always. The copy goes to the candidate model, which processes it normally — but its response is discarded before the user sees anything. The candidate's outputs are logged and compared against the live model's outputs.

Concretely: suppose you have a ranking model serving 10k QPS. You stand up a second serving fleet running the candidate. A routing layer (or a sidecar at the live fleet) clones each request, sends one to production, forwards the other to shadow, drops the shadow response, but records both scores. You then run offline comparisons: distribution of scores, top-K overlap, tail-latency of the candidate, error rate.

A/B testing for ML systems has more pitfalls than A/B testing for UI changes, because the treatment (a model) affects outputs in subtle, correlated, and sometimes delayed ways. Here are the components you must get right.

Randomization unit. The unit of randomization must be chosen carefully. For most ML experiments, it is the user (or device). Randomizing by request is wrong for ranking/recommendation: the same user might get model A on one click and model B on the next, causing carry-over effects and undermining independence. Users must be consistently assigned throughout the experiment.

Standard A/B tests for ranking systems require large samples and long run-times because the metric (e.g., CTR on a results page) is noisy: a bad result at rank 3 might still get clicked if ranks 1 and 2 are great. Interleaving is a technique that gets the same statistical signal in roughly 100× fewer users.

How it works (Team Draft Interleaving):

1. For a single user request, run both models A and B, getting ranked lists L_A and L_B.
2. Build a combined list by alternating picks: flip a coin to decide which model picks first. Model A picks its top item not yet in the list, then model B picks its top item not yet in the list, repeat until you have enough items.
3. Show the interleaved list to the user. Record which model "owned" each item the user clicked.
4. If model B's items get more clicks, B wins this impression. Aggregate across thousands of impressions for a win-rate.

Why it is so much faster: each user impression is a paired comparison between the two models on identical context. The noise from "this user just isn't a clicker today" cancels out within the impression. A/B tests don't have this pairing — the two groups see different contexts and different users, adding variance. Interleaving can detect a 1% ranking improvement with days of traffic instead of weeks.

Once the A/B test clears, you ramp traffic from the A/B split to 100% treatment. This is typically done gradually: 50% → 75% → 100% over hours or days, watching system metrics at each step. The ramp is not an experiment; it is a controlled deployment with early-warning monitoring.

After reaching 100%, maintain a holdback group: a small fraction (1–5%) of users who still see the old model. This lets you continue computing the treatment effect long after the A/B test ended, catching long-term or novelty effects that the experiment window was too short to see. Holdback users are typically randomized at the device or account level and held stable for weeks or months.

When to retire the holdback: once the treatment effect has been stable for long enough that novelty is clearly not a factor and no late-appearing regressions have surfaced, typically 4–8 weeks for significant model changes. Retiring too early means you lose your comparison baseline. Retiring too late is wasteful (you are permanently serving a worse experience to some users) and the holdback group starts to diverge demographically.

At companies running thousands of simultaneous experiments, a few engineering patterns are mandatory.

Orthogonal experiment layers. Facebook, Google, and Netflix use a layered experiment framework where each experiment layer controls a different system (ranking model, UI, notifications, pricing). A user is assigned to one bucket per layer. Layers are designed to be orthogonal so experiments in different layers do not interact. This allows thousands of simultaneous experiments without mutual contamination.

Assignment service. A centralized service resolves which variant a user is in, given their user ID and the experiment definition. It must be fast (sub-millisecond), deterministic (same user always gets same variant), and consistent across all services that need to know the assignment.

Metric computation pipeline. Raw event logs flow into an aggregation pipeline that computes per-experiment, per-metric statistics with confidence intervals. A platform-level significance test runs automatically. Engineers configure which guardrail metrics are blocking (must clear) vs. informational (logged but not blocking).

Power analysis tooling. Before launching an experiment, a power calculator estimates how many users and how many days are needed to detect the minimum effect the team cares about. Underpowered experiments waste time (run for two weeks, conclude "no signal," but actually the effect was real and just below detection).

A web server that starts returning 500s screams immediately. An ML model that quietly starts returning subtly wrong predictions may produce no errors at all — serving latency stays flat, HTTP 200s keep flowing, and the only signal is a slow drift in a business metric that might be blamed on seasonality for weeks.

Three root causes make ML monitoring unique:

• Behavior is learned from data, not code. A silent upstream schema change (a feature column renamed, a vocabulary expanded) can shift predictions without touching a line of model code.
• Ground truth arrives late or never. For a fraud model, you may not know if a transaction was fraudulent for days. For a recommendation model, "did the user enjoy this?" is inferred indirectly.
• Degradation is continuous, not binary. Quality erodes gradually; there is no crash, no stack trace.

The answer is a four-layer monitoring stack: each layer catches failures the layer above cannot.

What it measures: QPS, request latency (p50/p99/p999), error rates, GPU/CPU utilization, memory, queue depth, cache hit rate.

What it catches: hardware failures, network outages, code regressions, traffic spikes, OOM kills.

Example alerts:

• p99 latency > 200ms for 5 consecutive minutes → page on-call.
• error rate > 1% over 1 minute → page on-call.
• GPU utilization < 20% for 10 minutes → possible batch stall or worker crash.
• request queue depth > 500 → autoscaler lag or upstream surge.

Why it's necessary but not sufficient: system metrics can be perfectly green while the model silently serves garbage predictions. You need layers 2–4.

What it measures: schema validity, null rates, out-of-range values, and statistical distributions of each feature seen at serving time.

What it catches: upstream data pipeline changes (column dropped, encoding flipped), seasonal distribution shifts, sensor failures, vocabulary drift.

Example alerts:

• null rate for feature `user_age_bucket` jumped from 0.3% to 18% → upstream join broken.
• PSI for feature `query_length` > 0.2 over 24h window → distribution shift, investigate.
• value `device_type = "smart_tv"` appears in production but not in training vocabulary → schema drift; model will default to unknown embedding.
• feature `price_usd` exceeds training max by >3σ → extrapolation risk.

Implementation pattern: log a sample of serving requests (features + predictions, no labels) to a monitoring table. Run statistical tests hourly or daily against the training distribution as the reference.

What it measures: distribution of prediction scores, calibration, top-feature attributions (via SHAP or attention weights), confidence histograms.

What it catches: model drift that data monitoring misses (inputs look fine but the model's learned mapping is now wrong), calibration rot, unexpected feature dominance.

Example alerts:

• mean prediction score dropped from 0.42 to 0.29 over 48h → model underconfident; possible concept drift or upstream feature skew.
• top SHAP feature switched from `user_history_score` to `time_of_day` → model's reliance on signals has shifted; investigate data quality of `user_history_score`.
• fraction of predictions > 0.9 jumped from 5% to 22% → score distribution inflated; calibration broken or data leakage introduced.
• Expected Calibration Error (ECE) > 0.05 → predictions no longer match empirical frequencies; downstream thresholding will be wrong.

Why it's powerful: you can monitor model signals in real time without any labels. Score distribution shifts are often the earliest detectable signal of a problem, hours before business metrics move.

What it measures: click-through rate, conversion rate, session length, revenue per session, user-reported errors, thumbs-up/thumbs-down rates. These are the metrics the business actually cares about.

What it catches: failures that slip through all lower layers — subtle recommendation quality degradation, a calibration bug that doesn't affect the score distribution but does affect downstream ranking.

Example alerts:

• 7-day CTR rolling average dropped > 5% relative → open incident, compare with score distribution shift.
• add-to-cart conversion fell > 8% vs same-day-of-week 4 weeks ago → paged to ML + product teams jointly.

Why it's the last resort, not the first line: business metrics are noisy (seasonality, product changes, external events) and move slowly. A 2% CTR drop may take days to reach statistical significance. By the time a product metric fires, users have already been affected for hours. Layers 2 and 3 should catch problems faster.

The word "drift" is used loosely in interviews. Interviewers reward candidates who distinguish the three types precisely — they have different causes, different signals, and different fixes.

Data drift example — e-commerce search: Your search ranking model was trained mostly on desktop users. Mobile traffic share grows from 15% to 60% over a quarter. Query length, session duration, and scroll behavior all shift. P(Y|X) (which items are relevant given a query+user) hasn't changed, but the distribution of queries X has. The model wasn't trained to handle this region of input space well. Fix: retrain on current traffic mix.

Concept drift example — fraud detection: A new fraud ring adopts a technique your model has never seen: they mimic legitimate purchase patterns by warming accounts for 30 days before striking. The relationship between account-age features and fraud probability (P(Y|X)) has fundamentally changed — old signal "aged account = safe" is now corrupted. Fix: retrain with fresh labels from the new fraud pattern; feature engineering to capture the new signature.

Label shift example — disease screening: A COVID diagnostic model trained during a wave (base rate 15%) is deployed during a low-prevalence period (base rate 0.5%). Even if P(symptoms | disease) is identical, the model's calibration is badly wrong — it will over-predict. Fix: re-calibrate posterior probabilities using importance weighting by the new base rate.

Detecting drift requires a number that summarizes how far two distributions have moved. The two workhorses are Population Stability Index (PSI) and KL divergence.

Worked PSI calculation — feature "query_length_bucket":

Suppose query length is bucketed into 3 bins: short (≤3 words), medium (4–8), long (>8). Training distribution and last week's production distribution:

PSI = 0.093 — below the 0.1 threshold, so this feature is stable. If next week the long bucket shrinks further to 0.04, PSI would exceed 0.2 and trigger an alert.

• Upstream schema change: a column is renamed, a categorical encoding is reordered, a nullable field starts returning nulls for a new region. Model continues serving — it just silently receives wrong inputs.
• Feedback loops: a recommendation model influences user behavior, which generates training data that reinforces the model's existing biases. The model gets better at predicting what it already served, not what users actually want. Coverage collapses. Quality erodes.
• Seasonality: a model trained on summer traffic serves winter traffic with different purchase intent, query patterns, and user mix. Nothing broke; the world changed.
• External events: a pandemic, an election, a viral trend — any sudden shift in the world that was not in training data can instantly invalidate a model's learned associations.
• Training data aging: even without external events, the world drifts slowly. A model trained 18 months ago on user behavior profiles users that no longer exist at the same rates.

The most convenient measure of model quality — actual accuracy — often cannot be computed in real time because labels arrive late or not at all.

• Fraud: a chargeback may arrive 30–90 days after the transaction. You can't know precision/recall for last week's scores until next quarter.
• Recommendations: "did the user enjoy this?" is never directly observed. Engagement (click, watch-time) is a proxy, but not the same thing.
• Ads: post-click conversion may be measured days later, long after the ad impression.
• Search: relevance judgments are collected via human rater programs on a slow cadence.

The engineering response — proxy metrics: choose metrics that are observable fast and historically correlate with true label quality:

• Score distribution shifts (Layer 3) — available immediately, no labels needed.
• Short-term engagement proxies (immediate click vs 30-day purchase) — fast and partially informative.
• Human evaluation panels on a sample — weekly, but labeled.
• Partial label windows — for fraud, use chargebacks available within 7 days as a leading indicator even though the full 90-day window is more complete.

The key discipline: establish the historical correlation between your proxy and your true metric during a period when you had both, then trust the proxy in production. Monitor for proxy–truth correlation drift too.

A production monitoring stack for an ML system typically consists of these interacting components:

Sampling rate decisions: logging 100% of features at high QPS is prohibitively expensive. A common pattern: log metadata (prediction, model version, latency) at 100%, log full features at 1–5%, and log features for flagged/anomalous cases at 100%. The sampled 1–5% is sufficient for distribution statistics if QPS is high enough.

Suppose you run a social feed with 100 million candidate items and a 100 ms latency budget. Your heavy ranker is a 50-million-parameter neural network. How many FLOPs does one forward pass cost?

A rough rule: a forward pass through an N-parameter dense model costs approximately 2N FLOPs per sample (one multiply-add per parameter per input token, counted as 2 ops).

Scoring all 100 million items:

A powerful serving GPU delivers roughly 300 TFLOP/s = \$3 \times 10^{14}$ FLOPs/s under real conditions. So the wall-clock time would be:

33 seconds for a 100 ms budget. You're off by a factor of 330. The only solution is to make the expensive ranker score far fewer items — 200 to 1000 instead of 100 million. That is the retrieval stage's job.

The latency budgets are additive. A 100 ms SLO might allocate: 10 ms retrieval + 5 ms light rank + 50 ms heavy rank + 10 ms re-rank + 25 ms feature fetch/network overhead.

The two-tower model (also called dual encoder) is the dominant neural retrieval architecture. Here's the intuition:

• User tower: takes user features (ID, history, context) → outputs a dense vector u ∈ ℝ^d.
• Item tower: takes item features (ID, content, metadata) → outputs a dense vector v ∈ ℝ^d.
• Score: score = u · v (dot product). Optionally L2-normalized for cosine similarity.

Why dot-product specifically? Because dot-product (and cosine) similarity can be computed with Approximate Nearest Neighbor (ANN) indexes. If you used a cross-product of features between user and item (like a full interaction model), you can't pre-index the items — you'd have to score every item fresh for every user. With dot-product, you pre-compute and index all item embeddings offline, then at serve time you compute the user embedding once and do an ANN lookup. That's what enables sublinear retrieval.

In-batch negatives: During training, for each (user, positive-item) pair, the model uses all other items in the same mini-batch as negatives. With a batch size of 1024, each training example gets 1023 free negatives. This is efficient but introduces a bias — popular items appear frequently as negatives, so the model is implicitly penalized for scoring popular items highly. Corrections: hard negative mining (deliberately sample items the model currently ranks high but are not positives) and popularity debiasing in the loss.

Once you have item embeddings, you need to find the top-K most similar to a query vector without scanning all 100 million items. That's the ANN problem. Three dominant approaches:

IVF (Inverted File Index)

Step 1: Cluster all item vectors into C clusters (e.g., C = 4096) using k-means. Step 2: Store each item in its cluster's inverted list. At query time: (1) compute distance from the query vector to all C cluster centroids — cheap, C centroids not 100M items; (2) pick the top nprobe nearest clusters (e.g., nprobe=32); (3) exhaustively score only the items in those clusters. If you have 100M items in 4096 clusters, each cluster has ~24K items. With nprobe=32 you scan 32×24K = 768K items instead of 100M — a ~130× speedup. Recall vs speed knob: increase nprobe → higher recall, slower.

HNSW (Hierarchical Navigable Small World)

Builds a multi-layer graph where each node (item embedding) connects to its nearest neighbors. The top layer is sparse (long-range links), lower layers are progressively denser. Search: start at the top layer, greedily walk toward the query, descend to find closer neighbors at each layer. Like a skip-list but in high-dimensional space. HNSW achieves very high recall at very low latency — often the best out-of-the-box — but requires storing the graph edges, which adds significant memory on top of the vectors.

PQ (Product Quantization)

Compresses vectors for memory savings. Split a 128-dim vector into 8 sub-vectors of 16 dims each. Quantize each sub-vector to one of 256 cluster centroids. Now represent each item vector as 8 bytes instead of 128×4 = 512 bytes — a 64× compression. Distance is approximated by looking up precomputed sub-vector distances. Usually combined with IVF: IVF narrows the candidate list, PQ provides compressed scoring inside each cluster.

Real systems combine multiple retrieval sources to maximize recall coverage. Each source captures a different signal:

• Follow graph: Items posted by accounts the user follows. High precision for social feeds; low recall (limited to who they follow).
• Collaborative filtering (item-item): "Users who engaged with items you engaged with also liked these." Classic ALS embeddings or SLIM. Strong for discovery.
• Popularity / trending: A simple baseline that's surprisingly hard to beat for new users (cold-start) and breaking news. Time-bucketed: trending-1h, trending-24h, trending-7d.
• Real-time session signals: Items similar to what the user is engaging with RIGHT NOW. Requires near-real-time embedding updates — expensive but high relevance for long sessions.
• Contextual retrieval: Query-driven (search-like): encode user's explicit query and retrieve semantically matching items.

The union is deduplicated (by item ID) and size-capped before entering the light ranker. Typical union size: 5K–50K items after dedup.

Here's the most common and expensive mistake in ranking systems: you score items at serve time using live feature values, but you log only the item ID and the user's eventual action (click or not). Later, when building the training dataset, you recompute the features for each logged impression.

Why this is catastrophically wrong: Feature values change over time. A feature like "item like count" or "author follower count" computed one week later is not the same value the model saw when it made the prediction. The model's score was based on the old value; the label (click) corresponds to the old value; the recomputed feature has a different value — you're training on mismatched input/label pairs. This is a form of training-serving skew that's especially insidious because the feature exists — it just has the wrong value.

The fix: log features at scoring time. When the model scores an item for a user, write the exact feature vector used (or a pointer to a feature snapshot) alongside the impression. Training retrieves these logged features, not recomputed ones. This is called point-in-time correct feature logging, and it's non-negotiable in production ranking.

Storage implication: if you serve 10M impressions/day and each has a 1KB feature vector, that's 10 GB/day of feature logs. Use a columnar store (Parquet) and retain for 30–90 days (enough for delayed label collection). Compress with Zstd.

The label sparsity problem: CTR (click-through rate) data is abundant — you see a click or not for every impression. But conversion (purchase, long watch, share) is rare — maybe 1 in 500 impressions converts. Training a model purely on conversions means 499/500 labels are negative; the model has almost no positive signal for a behavior you care about most.

The single-objective problem: If you optimize only CTR, the model learns to recommend clickbait — thumbnails and titles engineered to get the tap, but the content disappoints. Users click, don't watch, and churn. You want to simultaneously optimize click AND watch time AND share AND explicit like — but these objectives sometimes conflict (clickbait is high-CTR, low-watch-time).

Multi-task learning (MTL) solves both: Share the feature representation across tasks. A single bottom network learns rich item/user representations; task-specific heads branch off the top. Positive examples for watch time (which are sparse) still train the shared representation, benefiting CTR prediction — and vice versa.

Shared bottom architecture (plain):

Input features
     |
[Shared bottom MLP]
     |
  -------
  |     |     |
[CTR] [WatchTime] [Share]
head   head        head
  |     |          |
logit  regression  logit

This is the simplest MTL design. It assumes all tasks benefit equally from sharing. If tasks are too dissimilar (e.g., CTR for news vs. CTR for video), the shared bottom can hurt each task by averaging conflicting gradient signals — this is called negative transfer.

MMoE (Mixture of Experts for Multi-task) — plain-words:

MMoE (Google, 2018) addresses negative transfer by replacing the shared bottom with K expert sub-networks (e.g., K=8). Each task gets a gating network that produces a soft weighted sum over the K experts. Tasks that are similar will learn gates that route to the same experts; dissimilar tasks can route to different experts. This gives each task the expressiveness of a task-specific model while still allowing positive transfer on shared experts.

In practice: MMoE significantly outperforms shared-bottom when tasks have mixed correlation (some share signal, some don't). The gating mechanism adds only K small networks per task — negligible parameter overhead vs the experts themselves.

The heavy ranker outputs a score (or multiple scores in MTL). To rank items, you need to combine them into a single value and sort. A typical combination formula:

Why calibration is REQUIRED before this combination:

Model outputs are typically raw logits or sigmoid-activated scores — they are scores, not probabilities. A CTR head might output 0.8 for an item; a like head might output 0.3 for the same item. Does 0.8 CTR + 0.3 like = 1.1 mean anything? Only if both scores are calibrated probabilities (i.e., a score of 0.8 means the event occurs 80% of the time).

Without calibration, the scales of different scores are arbitrary. The model might produce CTR scores in [0.1, 0.9] but like scores in [0.001, 0.05] simply because the base rates differ (likes are rare). Adding or multiplying these uncalibrated scores distorts the combination.

Additionally, calibration is required whenever scores are thresholded (e.g., "only show items with predicted CTR > 0.05") or used in bidding systems (ad auctions use predicted click probability to determine bid prices — if your probability is 2× too high, you overbid by 2×).

Platt scaling: Fit a logistic regression on validation data: P(y=1) = sigmoid(a * score + b). Learn parameters a, b on held-out labeled data. Fast, often sufficient.

Isotonic regression: A non-parametric monotone regression that can correct more complex miscalibration shapes. Requires more data. Use when you see systematic over/under-confidence in certain score ranges (check calibration plots: plot mean predicted probability vs actual event rate in score buckets).

Click logs confound two things: how good an item was, and where it was shown. Rank 1 gets seen; rank 20 barely exists. A model trained naively on clicks learns "rank 1 = clickable" — it learns the OLD ranker's choices, not item quality. Eye-tracking and randomization studies put rank-1 vs rank-10 examination rates at 5-10× apart, which is larger than most real quality differences.

The heavy ranker scores items independently; the final list is assembled with cross-item logic:

• Diversity (MMR): greedily pick next item by λ·score − (1−λ)·max_sim(to already picked) — stops five near-identical videos from sweeping the top of the feed.
• Business rules: publisher caps, integrity filters, contractual slots, freshness quotas — applied as constraints, not score hacks, so they're auditable.
• Exploration slot: reserve a small probability of a high-uncertainty item; this is the funnel's oxygen supply (cold start, feedback-loop relief).

Why not bake diversity into the model? Because it's a property of the SET, not of any item — an item's marginal value depends on what's already above it. Set-level objectives belong in the assembly step where the set is visible.

Imagine a recommendation system that recomputes user profiles once per day at 3 AM. At 8 PM, a user opens your app and spends 40 minutes watching five consecutive cooking videos — boiling pasta, pan sauces, homemade bread. Their declared intent is now crystal clear: show me more cooking content. But the model's user vector was computed 17 hours ago and reflects a generic "watches drama and cooking occasionally" profile. Every recommendation the system makes during this session is based on stale evidence.

The cost of staleness is not uniform. For a slow-changing feature like "user's all-time favorite genre," a 24-hour lag is usually harmless. For a fast-changing signal like "what the user just watched in the last 10 minutes," a 24-hour lag means the feature is effectively random noise. Intent changes at the session level — within one sitting — and batch systems are blind to it.

Worked numbers. Suppose a user watches a cooking video at 8:05 PM. Without streaming features, the recommendation model is blind to that event until the next batch run at 3 AM — a 7-hour lag. If the user watches the next video at 8:10 PM, you missed a 5-minute window to personalize the next recommendation. Across 10M daily active users who each have 3 such session-level intent shifts per session, that's 30M missed personalization opportunities per day.

A streaming feature pipeline captures user events in near-real-time and makes them available at serve time. The canonical architecture has four stages:

User action (click/watch/like)
        |
   [Event bus — Kafka topic]
        |
   [Stream processor — Flink]
   - windowed aggregates: last-5-min clicks, last-30-min genres
   - sessionization: group events by user+session
        |
   [Online feature store — Redis / DynamoDB]
   key: user_id  value: {genres_30m: [cooking:4, drama:1], ...}
        |
   [Model serving — reads online store at request time]

Each stage adds latency — the freshness budget. Define end-to-end freshness as the time from "event occurs" to "feature is visible to the ranking model." Budget each stage:

Total freshness budget example: With a 10-second sliding window, end-to-end freshness ≈ 200 + 300 + 10,000 + 10 + 3 ≈ 10.5 seconds. The user's binge-watching signal is visible to the model within 10 seconds of the event. Compare to 7 hours for a batch pipeline — a 2,500× improvement in freshness.

Window type choice matters. A tumbling window (non-overlapping intervals: 0–60s, 60–120s, …) is cheap but introduces up to one full window of latency. A sliding window (e.g., every 10s, compute last 60s) is more expensive (overlapping computations) but reduces latency to the slide interval. For session-level personalization, sliding windows of 30–60 seconds are a practical sweet spot.

Training/serving consistency. You train on historical data. If training uses batch-computed features but serving uses streaming features, you have training-serving skew. Solutions: (a) Lambda architecture: keep both; batch features for training, streaming features for serving, accept a small skew; (b) Kappa architecture: treat the Kafka log as the source of truth, replay it to build training features the same way the serving pipeline does. Lambda is simpler operationally; Kappa is more consistent but requires Kafka retention of months of events.

The richest representation of a user's current intent is their raw interaction sequence: "watched video A, then B, then C, then queried 'pasta bolognese'." A transformer encoder over this sequence can produce a user embedding that captures temporal order and context shifts — far more expressive than any aggregate feature.

Plain-words architecture. At each request, the system retrieves the user's last N interactions (item IDs, timestamps, action types) from a sequence store. These are embedded, position-encoded, and passed through a shallow transformer encoder (2–4 layers is typical for latency reasons). The output CLS token or mean-pooled output becomes the dynamic user embedding, which is then used as a feature in the ranking model.

# Conceptual sequence encoder at serve time
history = store.get_user_history(user_id, max_len=64)   # last 64 items
item_embs = embedding_table[history.item_ids]            # shape (64, d)
pos_enc   = positional_encoding(history.timestamps)
x = item_embs + pos_enc                                  # (64, d)
user_vec  = transformer_encoder(x).mean(dim=0)          # (d,)
# user_vec replaces or supplements static user features in ranker

Why sequences beat aggregates. An aggregate feature like "user's top genre in the last 30 minutes = cooking" loses the sequence order. The transformer can capture: "user watched action, then action, then cooking — they may be transitioning topics." It can also model repetition differently from exploration. These signals are simply not expressible in aggregate form.

Cold start is the problem of making good recommendations when you have zero or near-zero historical data. It has two flavors that require different solutions.

New user cold start. A user signs up today. You know nothing about their preferences. What do you serve?

New item cold start. A creator uploads a new video today. You have zero engagement data. How do you rank it?

A decoder-only transformer generates text by extending a sequence one token at a time. At step t, the model receives the full context (prompt + all tokens generated so far) and produces a probability distribution over the vocabulary. One token is sampled, appended to the context, and the process repeats.

Concretely, for each transformer layer, the input sequence of length t is projected into query (Q), key (K), and value (V) matrices. Attention is then computed:

The critical observation: the new token only needs to attend to past tokens, but to do so it still must have access to the K and V representations of every past position. That is the root of all the complexity that follows.

Suppose we naively regenerate everything on every step. To produce token t, we run a full forward pass over all t tokens. For a model with L layers, hidden dimension d, and H attention heads each of size d_k = d/H:

• The Q·Kᵀ matmul in one layer costs 2 · t² · d FLOPs (across all heads).
• The Attn·V step costs another 2 · t² · d FLOPs.
• The four projection matrices (Q, K, V, O) each cost 2 · t · d² FLOPs.
• FFN (two linear layers, typical expansion 4×) costs ~16 · t · d² FLOPs.

Attention FLOPs dominate at long contexts. Summing over all tokens generated (1 through n):

The key insight: K and V for position i depend only on the token at position i — they do not change as we generate future tokens. So we can compute K_i and V_i once, store them in a buffer (the KV cache), and reuse them for every future step.

At step t, we only compute Q for the new token (shape 1 × d_k), then dot it against the cached K (t × d_k). The Q·Kᵀ matmul is now 1 × t rather than t × t: one row instead of t rows.

The improvement is a factor of n/6 in attention FLOPs. For n=1000 that is roughly 167× fewer FLOPs just in attention.

Let's do this for Llama-2-7B-class architecture (public numbers): 32 layers, hidden dim 4096, 32 attention heads, head dim 128, GQA with 32 KV heads (standard MHA for this size). We generate n = 1000 output tokens after a prompt of negligible length.

Without KV cache (attention FLOPs only, one token at a time):

That is ≈ 175 TFLOPs in attention alone for 1000 tokens, not counting projection and FFN.

With KV cache (attention FLOPs):

The projection and FFN costs are O(n · d²) and identical with or without the cache (we always run them once per token). For context: those cost roughly 2 × 32 × (4 × 4096² + 2 × 4096 × 11008) × 1000 ≈ 11 TFLOPs. So without the cache, attention dominates overwhelmingly; with the cache, projections and FFN dominate at 1000 tokens — a completely different workload profile.

Storing the KV cache costs memory proportional to sequence length and batch size. Every layer needs to store a K tensor and a V tensor:

Single sequence, 4096 context, fp16 (2 bytes):

Step by step: 32 layers × 32 KV-heads × 128 head-dim = 131,072 floats per position. Times 2 (K and V) = 262,144 floats. Times 2 bytes (fp16) = 524,288 bytes per token. Times 4096 tokens = ≈ 2 GB per sequence.

Now scale to batch size 64:

The formula scales linearly with both sequence length and batch size. At 32k context (common in production today), a single sequence's KV cache is already 16 GB. This is not a corner case — it is the daily reality of serving frontier models.

Multi-Head Attention (MHA): each attention head has its own K and V projections. H_kv = H_q. Most expensive in memory.

Multi-Query Attention (MQA): all query heads share a single K and a single V head (H_kv = 1). Reduces KV memory by H_q×. Quality can degrade slightly.

Grouped-Query Attention (GQA): query heads are split into G groups, each group sharing one K/V head (H_kv = G). Llama-3-8B uses G=8 (32 query heads, 8 KV heads). For our 7B example with GQA-8:

Every LLM request has two distinct phases, and they hit the hardware in fundamentally different ways.

Prefill: The prompt tokens (say, 512 tokens) are all known upfront. We process them in one forward pass with full parallelism: the Q·Kᵀ matmul is (512 × d_k) × (d_k × 512) — a fat matrix multiply. This has high arithmetic intensity (FLOPs per byte moved). The GPU's tensor cores are fully utilized. The result of prefill is: (a) the KV cache is populated for all prompt positions, (b) the first output token is produced. The user-visible metric is TTFT (Time To First Token).

Decode: After the first token, we enter autoregressive decode. Each step processes exactly one new token. The Q·Kᵀ matmul is now (1 × d_k) × (d_k × t) — a matrix-vector product. Arithmetic intensity collapses. For each weight matrix W (shape d × d), we do 2d² FLOPs but read 2d² bytes (fp16). Arithmetic intensity = 1 FLOP/byte — far below the H100's ~160 FLOP/byte compute-to-bandwidth ratio. The bottleneck is memory bandwidth, not compute. The GPU sits mostly idle waiting for weights and KV cache to stream from HBM. The user-visible metric is TPOT (Time Per Output Token).

TTFT (Time To First Token): How long from request submission to the first byte of response. Dominated by prefill compute (and queuing). Users perceive this as "loading time". A streaming UI can mask short TTFT even for long responses — so TTFT matters more for chatbots than for batch jobs.

TPOT (Time Per Output Token): Average inter-token latency during decode. Drives perceived "streaming speed". Too slow → text dribbles and users bail. End-to-end latency for a response of n tokens is roughly:

For a 500-token response with TTFT=200ms and TPOT=20ms: total ≈ 200 + 499×20 ≈ 10.2 seconds. The UX impact of TPOT dominates for long outputs — halving TPOT matters more than halving TTFT.

Early LLM serving systems (pre-2023) used static batching: collect a batch of B requests, run them together until all of them finish, then accept the next batch. This seems sensible — until you realize that sequence lengths vary wildly.

Imagine a batch of 8 requests. Seven finish at 80 tokens. One request (a long-form essay) runs to 2000 tokens. Under static batching:

• Steps 1–80: all 8 sequences are active. GPU is used well.
• Steps 81–2000: only 1 sequence is active. The other 7 slots sit empty — consuming allocated KV memory but doing no work. GPU utilization: 1/8 = 12.5%.
• New requests queue outside even though 7 GPU slots are free.

The wasted GPU-steps in this toy example: (2000 - 80) × 7 = 13,440 idle token-slots. In a real serving fleet with variable-length traffic, utilization can fall below 30% with static batching.

The Orca paper (2022, and independently vLLM 2023) identified the fix: instead of treating a batch as the unit of scheduling, make the iteration (decode step) the unit. After every single decode step, the scheduler checks: did any sequence just generate an EOS (end-of-sequence) token? If so, immediately evict that sequence from the batch and admit a waiting request.

Under continuous batching, the 8-request example above works like this:

• Step 80: sequences 1–7 finish. They are evicted; 7 new requests admitted.
• Step 81 onward: the long request runs alongside the 7 new ones. GPU stays near-fully utilized.

The throughput gain depends on the length variance of requests. In practice, continuous batching achieves 2–10× higher throughput vs static batching on real traffic distributions, with the high end seen when traffic is a mix of very short and very long requests.

The fragmentation problem. Even with continuous batching, naive KV cache management has a serious memory waste problem. When a request arrives, the system doesn't know how long the output will be. A common approach: pre-allocate the maximum possible length (say, 4096 tokens) as a contiguous block. But most requests finish in a few hundred tokens. The reserved-but-unused tail of the block is wasted. Worse: contiguous allocation means the allocator must find a free contiguous chunk — as memory gets fragmented, allocation fails even when total free bytes would suffice.

In a system with maximum sequence length 4096 and requests averaging 256 output tokens, naive pre-allocation wastes up to 16× the actually needed KV memory — severely limiting the batch size that fits in GPU memory.

The insight from vLLM (2023): apply the same trick that operating systems use for RAM — paging. Instead of requiring contiguous physical memory, divide the KV cache into fixed-size blocks (pages) of, say, 16 tokens each. Use a block table (analogous to a page table) to map each sequence's logical KV positions to physical block addresses. A sequence grows by allocating new blocks on demand; no contiguous pre-allocation needed.

The bonus: KV sharing. When multiple requests share a common prefix — e.g., a system prompt, a few-shot template, or a document being queried by many users — their KV blocks for that prefix are physically identical. With paged attention, those blocks can be shared (copy-on-write) rather than duplicated. This is prefix caching.

The bottleneck it fixes. Decode is memory-bandwidth-bound at batch-1 (or small batch). The GPU loads all model weights each step to produce one token. Idea: what if we could produce multiple tokens per big-model forward pass?

The mechanism. Keep a small, fast draft model (or a draft mechanism — see below) alongside the large target model. At each step:

1. The draft model autoregressively generates k candidate tokens (γ = k is typically 3–7). This is cheap because the draft model is small.
2. The target model runs a single forward pass over the current context + k draft tokens in parallel. Because all k+1 positions are known, this is a prefill-like parallel pass — not sequential decode.
3. The target model produces probability distributions at each of the k+1 positions. These are used to accept or reject each draft token via a speculative rejection sampling procedure.
4. If all k draft tokens are accepted, we gain k+1 tokens from one target pass. If the first draft token is rejected, we recover 1 token from the target's corrected distribution. On average, accepted tokens per step = γ̄ ∈ (1, k+1).

The wall-clock speedup is:

In practice, a draft model that is 10–20× smaller than the target, with a token acceptance rate α ≈ 0.7–0.9 on typical text, gives 2–3× decode throughput for memory-bandwidth-limited scenarios.

The draft proposes token x with probability q(x); the target computes p(x) in its single verification pass. Accept x with probability min(1, p(x)/q(x)); on rejection, resample from the residual distribution norm(max(0, p−q)). Summing the two paths: P(emit x) = q(x)·min(1, p(x)/q(x)) + P(reject)·residual(x) = exactly p(x). The output is provably the target model's distribution — speculation buys speed, never quality, which is why it's a pure serving optimization you can enable without re-running evals.

Mixing prefill and decode in one batch creates interference: a 32k-token prefill is a multi-second compute monster, and every decode request batched with it stalls — TTFT for one user destroys TPOT for sixty. Two escalating fixes:

What breaks without either: p99 TPOT spikes whenever a long-context request arrives — the classic "our chat got janky after we launched document upload" incident.

A chat completion is not one latency number — it is a stream. Two numbers describe the user experience:

Why streaming changes everything: people read at roughly 250 words/min ≈ 4–5 words/s ≈ 6–8 tokens/s. If you stream at 20 tokens/s (TPOT = 50 ms), you outrun the reader ~3× and the answer feels instantaneous even though the full 500-token completion takes 25 seconds end to end. Without streaming, that same request is a 25-second blank screen. Same backend cost, wildly different perceived latency.

Typical chat targets (2025-era, p90): TTFT ≤ 300–500 ms for short prompts (long prompts get a budget that scales with prompt length, since prefill work is linear-ish in it), TPOT ≤ 50 ms (≥ 20 tok/s). Code-completion products are far stricter on TTFT (≤ 150 ms — the developer is mid-keystroke); batch/offline pipelines have no TPOT target at all and are optimized purely for throughput, i.e., cost.

The only formula you need, then a fully worked example. Cost per token is just GPU rental rate divided by token throughput:

Setup: a 70B dense model served in bf16 on one replica of 4×H100 (tensor-parallel), cloud price \$3/GPU-hr → the replica costs \$12/hr.

Step 1 — decode throughput ceiling from bandwidth (the physics check). Decode is memory-bound: every decode step must stream all 140 GB of weights (70B params × 2 bytes) through HBM. Aggregate bandwidth of 4 H100s ≈ 4 × 3.35 ≈ 13.4 TB/s. So one forward pass takes at least 140 / 13,400 ≈ 0.0104 s ≈ 10.4 ms → at most ~96 forward passes per second, regardless of batch size (KV reads and comms push this up further).

Step 2 — batch size multiplies tokens, not passes. At batch 1: ~96 tok/s ideal; realistically ~50 tok/s after attention/KV reads, kernel and TP-communication overhead. At batch 48 with continuous batching: each pass emits 48 tokens → ideal 48 × 96 ≈ 4,600 tok/s; take a realistic 4,000 tok/s. Note TPOT barely moved (each sequence still sees one pass per token, now ~12 ms instead of 10.4) — batching is nearly free throughput until you saturate compute or KV memory.

Step 3 — dollars.

• Batch 48: 4,000 tok/s × 3600 = 14.4M output tokens/hr → \$12 ÷ 14.4 = \$0.83 per 1M output tokens.
• Batch 1: 50 tok/s × 3600 = 180k tokens/hr = 0.18M → \$12 ÷ 0.18 = \$66.7 per 1M output tokens.

Same hardware, same model: batching is an 80× cost difference (4,000/50 = 80). This single calculation is why continuous batching (ch18) is non-negotiable and why "GPU cost" questions are really "utilization" questions.

Step 4 — why input tokens are cheaper than output tokens. Prefill processes the whole prompt in parallel and is compute-bound, so the same replica might sustain ~40,000 prompt tokens/s. That's 40,000 × 3600 = 144M tokens/hr → \$12 ÷ 144 = \$0.083 per 1M input tokens — about 10× cheaper than the \$0.83 output figure. This is exactly why every commercial API prices input tokens several times cheaper than output tokens: the asymmetry is physics (parallel compute-bound prefill vs. serial bandwidth-bound decode), not marketing.

Step 5 — does batch 48 even fit? KV per token for a 70B-class model (80 layers, 8 GQA KV-heads, head_dim 128, bf16): 2 × 80 × 8 × 128 × 2 bytes = 327,680 B ≈ 320 KB/token. At 4k context: 4096 × 320 KB ≈ 1.3 GB per sequence → batch 48 ≈ 63 GB of KV. The replica has 4 × 80 = 320 GB HBM, minus 140 GB weights = 180 GB free. Fits with headroom. The napkin closes.

A fleet of replicas needs a router that understands KV caches:

Context length is a cost multiplier hiding in plain sight. For a 13B GQA model (40 layers, 8 KV heads, head_dim 128, fp16): per-token KV = 2 × 40 × 8 × 128 × 2B = 164KB. At 4k context that's 0.7GB per sequence; at 128k it's 21GB — one sequence per GPU, batch collapses, and the per-request cost rises roughly linearly in context while perceived value doesn't. Hence the menu: context caps by product tier, prompt compression/summarization, context caching (bill the cache, not recompute — why providers price cached input tokens ~10× cheaper), GQA/MQA and KV quantization (fewer bytes per token), and retrieval instead of stuffing (RAG as a cost decision, not just a quality one).

Before looking at the pipeline, understand the motivation — interviewers will ask "why not just fine-tune?" and you need a crisp answer.

A production RAG system has a fixed sequence of stages. Every stage can fail independently — this is the key insight for debugging. Study each stage as (a) what it does, (b) what breaks if it is wrong.

What it does: raw documents (PDFs, HTML, Confluence pages, Slack messages) are parsed, cleaned, and split into chunks that will become the retrieval units. Each chunk is embedded and stored as one vector.

Why chunking matters: embedding models have a token limit (typically 512–8192 tokens). A 50-page PDF cannot become one vector — it must be split. But how you split determines what the system can retrieve.

What it does: each chunk is passed through an embedding model to produce a dense vector (e.g., 768 or 1536 floats). The query at retrieval time is embedded with the same model. Similarity between query vector and chunk vectors drives retrieval.

The language gap: multilingual documents and English queries require a multilingual embedding model, or translated queries. Missing this produces near-zero recall on non-English content.

Embedding model versioning: if you update the embedding model, ALL existing vectors must be re-embedded. Mixing vectors from two model versions in the same index makes distances meaningless — a silent failure that degrades retrieval gradually as new documents use the new model.

What it does: vectors are loaded into an Approximate Nearest Neighbor (ANN) index so that at query time, the top-k most similar chunk vectors can be found without scanning all N vectors exhaustively.

At 1 million chunks of 768-dimensional float32 vectors, exhaustive search requires 1M × 768 = 768M multiplications per query — feasible for small corpora but slow for large ones. ANN indexes trade a small recall penalty for orders-of-magnitude speedup.

What it does: the user's query is embedded and the top-k most similar chunks are retrieved from the index. This is the stage that makes or breaks the system: if the right chunk is not retrieved, no amount of LLM cleverness can recover.

Dense retrieval alone fails on keyword queries. "What is the invoice number for PO-44921?" is a lookup, not a semantic question. Semantic similarity may rank "invoice processing workflow" higher than the chunk containing the actual number. Keyword search (BM25) handles this naturally because it matches exact terms.

Hybrid retrieval = dense + sparse + fusion. The production solution is to run both a dense ANN retrieval and a BM25 sparse retrieval, then fuse the ranked lists:

RRF is robust because it does not require normalizing scores across different retrieval systems — it only uses ranks. A document ranked #1 by BM25 and #3 by dense retrieval scores very high; a document ranked #50 by both scores very low.

What it does: ANN retrieval returns top-k chunks (e.g., k=50) quickly but with moderate precision. A re-ranker (cross-encoder) reads the query AND each candidate chunk together and produces a more accurate relevance score, then re-sorts to produce a shorter list (e.g., top-5) passed to the LLM.

Why two stages? A cross-encoder that reads query+document together is far more accurate than a bi-encoder that embeds them separately — but it is 10–100× slower because it cannot pre-compute document representations. The two-stage funnel gets the best of both: fast recall at k=50, precise ranking at top-5.

What it does: the top-k retrieved chunks (after re-ranking) are formatted into a prompt alongside the user's question and sent to the LLM. This stage is deceptively simple but has its own failure mode.

What it does: the LLM reads the assembled prompt and generates the answer.

RAG evaluation requires measuring each stage independently. One aggregate quality score is not enough — it hides which stage is broken.

Supervised Fine-Tuning (SFT) trains the pre-trained model on a curated set of (prompt, ideal response) pairs. The training objective is identical to language model pre-training — next-token prediction on the response tokens — but the dataset is hand-crafted or human-annotated, not scraped from the internet.

Why SFT? A pre-trained model is a document completer: given "The capital of France is", it predicts "Paris." Ask it "What is the capital of France?" and it may predict a follow-up question rather than an answer, because question-answering dialogs were a small fraction of its pre-training data. SFT shifts the distribution: the model learns to be a responder, not a completer.

Full fine-tuning updates all model weights. For a 7B-parameter model, that means storing and updating 7B gradients and optimizer states — roughly 112 GB at bf16 Adam (params 14 GB + grads 14 GB + Adam states 56 GB + activations). LoRA reduces this dramatically.

The core idea: instead of updating the full weight matrix $W \in \mathbb{R}^{d \times d}$, add a low-rank delta:

During fine-tuning, $W$ is frozen. Only $A$ and $B$ are trained. At inference time, $BA$ is added to $W$ — or equivalently, $W' = W + BA$ is computed once and the merged weight is used. This means LoRA adds zero inference latency if you merge before serving.

LoRA's most important operational consequence is not memory savings during training — it is that you can serve dozens of fine-tuned variants from one base model.

The problem it solves: if you fine-tune 50 customer-specific models (each fully fine-tuned), you need 50 × 14 GB = 700 GB of GPU memory to serve them simultaneously. With LoRA, each adapter is only 50–200 MB. You load the 14 GB base model once and swap LoRA adapters per-request.

Reinforcement Learning from Human Feedback is commonly described as an optimization algorithm. In systems terms, it is an orchestration problem: four separate model instances must run simultaneously, share gradients through a feedback loop, and do so at scale without running out of memory or deadlocking.

The fundamental systems challenge of RLHF is that inference infrastructure runs inside the training loop. During rollout:

1. The policy runs autoregressive generation (inference) on a batch of prompts — potentially hundreds of tokens per prompt.
2. The reference policy runs a forward pass on the generated sequences.
3. The reward model runs a forward pass to score each completed response.
4. PPO computes advantages using critic estimates and RM scores.
5. The policy and critic run backward passes and update their parameters.

Steps 1–3 are inference. Steps 4–5 are training. They alternate in every iteration. This means the system must handle:

• Memory pressure: four model copies + KV caches for generation + activations for backward. At 7B parameters × 4 models = 28B parameters worth of memory before activations — easily exceeding a single node's GPU memory.
• Throughput bottleneck: the policy generates responses token by token (slow, memory-bandwidth-bound). If you are waiting for 512 prompts each generating 256 tokens, the rollout phase dominates training time.
• Heterogeneous compute patterns: the generation phase wants memory-efficient autoregressive inference (paged attention, continuous batching); the update phase wants high-throughput matrix operations. Sharing GPU memory between these two patterns is non-trivial.

There are four levels of complexity in LLM-based systems, each a strict superset of the previous:

The jump from "single call" to "agentic" is not cosmetic. Each level introduces new failure modes, new infrastructure requirements, and new evaluation challenges. Understanding why — not just that — each level is harder is what separates a staff-level answer from a junior one.

The core problem in one sentence: if each step of an agent succeeds with probability p, the probability that a 10-step agent produces a fully correct result is p¹⁰.

Concrete numbers make this visceral. Suppose each step of your agent — LLM call, tool invocation, or parsing — succeeds 95% of the time. That sounds excellent for a single call. Now chain steps:

At p = 0.95 and n = 10: 0.95^10 ≈ 0.599. A 95%-per-step agent with 10 steps fails 40% of the time. That is unusable for most production tasks. The practical implications:

• Minimize steps ruthlessly. Every unnecessary step costs success probability, latency, and money. The best agent is the shortest agent that solves the problem.
• Drive p → 1 per step. Structured output schemas, tool input validation, retries with backoff, idempotent operations, and fallback paths all push p upward.
• Add checkpoints. After high-consequence steps, verify the result before continuing — analogous to transaction commit points. Don't wait until step 10 to discover step 3 silently produced garbage.
• Design for graceful degradation. What happens when the agent cannot complete? A partial answer delivered confidently beats a crash or an infinite retry loop.

Tool calling is the mechanism by which the LLM requests side-effects: reading a file, executing code, querying an API, writing to a database. The model emits a structured call (function name + arguments), the host system executes it, and the result is injected back into context. Getting this right is a systems problem, not a prompting problem.

A single chat completion is one event; an agent run is a trace — a tree of LLM calls, tool invocations, and retries. Production agent infra borrows distributed-systems tracing wholesale: every step gets a span (inputs, outputs, latency, cost, model version), traces are replayable (re-run the exact step sequence against a new model to regression-test), and aggregate dashboards track step counts, tool-error rates, loop detections (the agent retrying the same failing action), and cost per completed task. Evals shift from "is this answer good" to task completion rate on end-to-end scenarios, with per-step attribution when a task fails — was it retrieval, reasoning, or a tool error? Without traces, agent debugging is archaeology.

These are intentionally rough — ±2× is fine in an interview. Precision signals you memorized a spec sheet; order-of-magnitude fluency signals you understand the system.

An H100 can do 1000 TFLOP/s — but no production workload hits that. Three forces cut efficiency:

1. Memory-bound phases. During decode (one token at a time), the GPU is loading weights from HBM, not doing matmuls. An H100 with 80GB of params at 3 TB/s can load those weights in ~27ms — so decode throughput is bandwidth-limited, not compute-limited. Compute utilization in this phase may be 5–15%.
2. Communication overhead. Tensor-parallel all-reduces between GPUs take real wall-clock time. At 40GB model sharded across 8 GPUs with NVLink, the all-reduce per layer is ~5–10% of forward-pass time.
3. Stragglers, kernel launch overhead, bubbles. Pipeline-parallel bubbles alone can eat 10–20% throughput.

The industry benchmark is Model FLOPs Utilization (MFU): actual FLOPs used for the model divided by peak GPU FLOPs.

For capacity math, always use 40% effective utilization unless you have a measured number. That means 1000 TFLOP/s peak → 400 effective TFLOP/s for FLOPs-based estimates, and you scale bandwidth-bound estimates similarly.

Scenario: You serve a two-stage ranker. The heavy ranker is a 500M-parameter transformer that scores up to 500 items per request. You need 10,000 QPS at p99 < 100ms. How many H100s do you need?

Step 1: FLOPs per request. Each request scores 500 items. Each score is a forward pass through a 500M-param model. Inference FLOPs ≈ 2N per item (one pass, one token equivalent for a classification model).

Step 2: Total FLOPs/sec demanded.

Step 3: Effective FLOPs per GPU. H100 at 40% MFU = 400 TFLOP/s = 0.4 PFLOP/s effective.

Step 4: GPU count (raw).

Step 5: Latency sanity check. 13 GPUs handling 10k QPS = ~769 req/GPU/s. Each request needs 500 GFLOP. At 400 TFLOP/s effective: 500 GFLOP ÷ 400 TFLOP/s = 1.25ms compute time per request. p99 budget is 100ms — we have headroom. But wait: the 500 items are scored in batch — and we can batch across concurrent requests. If each GPU handles a batch of 64 requests simultaneously (32,000 items), the matmul is efficient. 13 GPUs × 64 batch = 832 in flight at once; at 769 req/s inflow, mean queue depth < 1 — we're fine.

Step 6: Add redundancy. Add 30% headroom for spikes and N+1 for zone failures → 13 × 1.3 × (2/1) ≈ 34 GPUs across two zones, ~5 nodes. State this explicitly.

Scenario: You want to train a 70B-parameter model on 1.5 trillion tokens (roughly Llama 2 70B scale). How many H100s do you need to train in 30 days?

Step 1: Total training FLOPs. The Chinchilla / PaLM result: training FLOPs ≈ 6ND.

Step 2: Time budget in seconds.

Step 3: Required FLOP/s across the cluster.

Step 4: Effective FLOP/s per H100. Peak 1000 TFLOP/s × 40% MFU = 400 TFLOP/s = 4 × 10¹⁴ FLOP/s effective per GPU.

Step 5: GPU count.

600 H100s = 75 eight-GPU nodes. At \$3/GPU·hr: 600 × \$3 × 24 × 30 = \$1.3M for the training run. This matches published estimates for 70B-scale models.

Step 6: Memory check. 70B params in bf16 = 140GB. One GPU = 80GB — so the model doesn't fit on one GPU. We need at minimum tensor parallelism across 2 GPUs. With ZeRO-3 (FSDP) across 600 GPUs, each GPU holds 140GB ÷ 600 ≈ 0.23GB of params — trivial. Activations + optimizer states are the real memory consumers; gradient checkpointing handles activations. This is consistent: large clusters use FSDP + pipeline-parallel for 70B training.

Scenario: A recommender system has a user embedding table: 500M users × 256-dimensional float32 embeddings. You also have an item table: 50M items × 128-dimensional float32. Design the memory layout.

Step 1: Raw sizes.

Step 2: Sharding decision. 512GB does not fit in a single machine's DRAM (typically 512GB–2TB on high-end servers — actually it might fit, but barely and with no room for model weights). More importantly, lookup throughput is the bottleneck: if we process 100k requests/sec and each looks up 1 user + 200 candidate items, that's 100k user lookups + 20M item lookups per second. A single DRAM bus cannot handle this.

Sharding strategy:

• User table (512GB): hash-shard by user_id across 8 CPU servers (64GB each). Each lookup is a network roundtrip: ~0.1ms with colocated RDMA; budget this into feature fetch SLA.
• Item table (25.6GB): replicate on every serving machine — it fits comfortably in DRAM, lookups are local, no coordination needed.
• Hot users: a "hot embedding cache" (Redis/Memcached) for the top 1% of users (5M × 256 × 4 = 5GB) keeps 80%+ of traffic served from in-process cache with < 1ms latency.

Step 3: Update frequency. User embeddings change daily (retrain), item embeddings change with new content (streaming). Design: nightly bulk rewrite for user table; Kafka → Flink → Redis for new-item embeddings within minutes.

These are strategy questions in interviews, not just cost questions. The right answer depends on workload predictability and the cost of interruption.

The "GPU-rich vs GPU-poor" frame: GPU-rich organizations (Anthropic, OpenAI, Google) can run long pretraining runs on reserved or owned clusters and amortize the fixed cost across many experiments. GPU-poor organizations must be aggressive about spot instances, smaller models, and weight-sharing — or buy inference via API at ~\$1–\$15 per million tokens (much higher unit cost but zero capex).

Hybrid strategy (most real companies): baseline serving on reserved GPUs (guaranteed availability), burst capacity on on-demand, batch/training on spot (fault-tolerant by design with checkpointing).

Every ML production incident belongs to one of five categories. Knowing which one you are in determines the right response playbook.

At scale, GPU failures are not rare events — they are scheduled occurrences. The math is simple and shocking.

MTBF (Mean Time Between Failures) for a GPU cluster:

A real H100 MTBF is 150,000–300,000 hours per device. But clusters also experience NIC failures, switch failures, power events, and software crashes. In practice, large training clusters (1k–10k GPUs) see a hard failure requiring checkpoint restart every 1–6 hours of wall-clock time.

Checkpoint cost math: A 70B model in bf16 = 140GB of weights. With Adam optimizer states (m, v in fp32) = 2 × 140GB = 280GB extra = 420GB total. Writing 420GB to NVMe at 7 GB/s takes 60 seconds. Writing to object storage at 6 GB/s (parallelized) takes ~70 seconds. This 1-minute interruption every 30 minutes = 3% overhead — acceptable.

Checkpoint frequency decision: If checkpoints cost time C and failures occur every F hours, the expected work lost per failure without a checkpoint is F/2 hours. With checkpoints every I hours, expected work lost = I/2. The optimal I = sqrt(2 × C × F) — but in practice, checkpoint every 15–30 minutes for long training runs.

Async checkpointing: Instead of pausing training, stream the checkpoint to host DRAM while the next training step continues. The GPU copies weights to CPU asynchronously; CPU writes to disk. This reduces the per-checkpoint training pause from 60s to ~5s (the PCIe copy time for a 420GB → 15GB weight slice). Modern frameworks (PyTorch FSDP, Megatron) support this natively.

Elastic training: Modern frameworks can resize the training job after a node failure — reduce the world size, redistribute shards, and continue from the last checkpoint without a full restart. This reduces downtime from "checkpoint + restart" (2–5 min) to "rejoin" (30s).

Serving reliability requires both redundancy (for hardware failures) and degradation plans (for overload and partial outages). The key insight: a degraded-but-serving system is nearly always better than a fully-down system.

Replica placement across zones: For a serving fleet, always spread replicas across at least 2 availability zones (3 is standard). A single-zone outage (rare but real — power, networking, cooling) should not take down serving. Minimum viable: N+1 replica count, where the N replicas in one zone can absorb full traffic if the other zone fails. This means running at ~50% utilization in normal conditions — a real cost, but unavoidable for 4-nines availability.

The graceful degradation ladder — commit this:

1. Full model serving (normal; all features, full reranking) → SLA: p99 < 100ms
2. Smaller/distilled model (if primary model overloaded or crashed) → quality drop ~5–10%, still personalized
3. Cached recommendations (precomputed batch results from the last hour, served from Redis) → stale but relevant
4. Popularity baseline (top-100 most popular items, zero personalization) → always available, degrades gracefully
5. Static fallback (a hardcoded editorial list) → last resort, never fails

Each level is a circuit-breaker trip. The serving layer checks primary health every 100ms; on 3 consecutive failures or latency p99 > 2× SLO, it trips to the next level and sets a 60-second hold-down before rechecking primary.

Load shedding: Under extreme overload (say, 5× normal traffic due to viral event), even the degradation ladder may be overwhelmed. Load shedding actively rejects a fraction of incoming requests with HTTP 429 (Too Many Requests) to protect the requests you do serve. The alternative — accepting all requests — causes queuing, latency explosion, and cascading timeouts where everything fails slowly. Fail fast, fail loud.

Retry storms: When a service is slow, clients time out and retry. If every client retries with the same backoff, retries arrive in a synchronized wave — exactly when the server is most stressed. Fix: exponential backoff with jitter (add a random 0–100ms offset to each retry interval). Use retry budgets: each client allows at most 20% of its requests to be retries; if the budget is exhausted, return an error rather than retrying.

Personal Identifiable Information (PII) is any datum that can identify a natural person — names, email addresses, phone numbers, IP addresses, credit-card numbers, medical record IDs, and subtler combinations (zip code + birthdate + gender uniquely identifies 87 % of Americans, per Latanya Sweeney's 1997 dataset). Internet-scraped corpora — the raw material for LLMs and many recommendation systems — are saturated with PII.

The risk is dual: training risk (the model memorizes PII and can regurgitate it on request) and compliance risk (GDPR Article 17, CCPA, and HIPAA all grant data subjects rights that are extremely hard to honour once the data is inside trained weights). Neither risk is theoretical: GPT-2 and GPT-3 have been shown to reproduce verbatim credit-card numbers and email addresses from training corpora.

The canonical pipeline runs three complementary passes over raw data before it enters training:

1. Regex + rule-based detection — fast, deterministic; catches SSNs (^\d{3}-\d{2}-\d{4}$), email addresses, phone numbers, credit-card patterns. False-negative rate is high on obfuscated or non-English PII.
2. NER-based detection — a fine-tuned sequence labeller (e.g., a BERT model trained on annotated PII corpora) catches PERSON, ORG, LOC entities in context. Higher recall but higher compute cost.
3. Heuristic deduplication — near-duplicate removal (MinHash / SimHash) reduces the probability that a rare PII string appears enough times to be memorised; memorisation risk scales sharply with repetition count.

Scrubbing strategies: redaction (replace with [REDACTED] or typed placeholder [EMAIL]) preserves document structure; synthetic replacement (swap a real name for a faker-generated one) preserves statistical patterns; document removal is the nuclear option used when a document is predominantly PII.

Scrubbing is necessary but not sufficient. Regulators increasingly require that you can demonstrate consent lineage: for every row in your training dataset, what data source did it come from, what terms of service or consent form governed that collection, and what downstream uses those terms permitted.

GDPR Article 17 grants individuals the right to erasure: if a person requests that their data be deleted, you must comply. For a database row, this is a DELETE statement. For a trained neural network, it is a fundamentally unsolved problem.

Why it's hard. Training compresses a dataset into billions of floating-point parameters. There is no pointer from a weight back to "the PII that influenced this weight." Retraining from scratch without the offending record is correct but prohibitively expensive — a 70 B parameter model trained on 2 T tokens takes ~2 × 10²⁴ FLOPs; a single erasure request cannot justify that cost.

Not everyone in an organisation should have access to every model or every feature. Two threats drive access control design:

• Data-use policy violations — a model trained on medical records should not be queryable by the ad-targeting team.
• Model exfiltration — weights are intellectual property; unrestricted access to model artefacts is a theft and compliance risk.

Interviewers at companies like Google, Meta, and Netflix do not expect you to have read the paper. They expect you to know the design space well enough to independently arrive at the same decisions. Use these studies to calibrate your intuition: "I'd do X because Y" — and then if you happen to know the real system did X, say so as a sanity check, not as the primary argument.

For each study the structure is: problem → scale → architecture → the 2–3 clever tricks → what to steal.

Problem: Rank a personalized feed of short or long videos for hundreds of millions of users in <100 ms, optimize for a mix of engagement signals (watch time, shares, likes), and do not implode the catalog into a filter bubble.

Scale: ~800 M daily active users (YouTube), corpus of hundreds of millions of videos, >1 billion watch events per day generating labels.

Architecture: A classic two-stage funnel. Retrieval: a two-tower model produces user and video embeddings; dot-product ANN retrieves ~500 candidates from ~800 M corpus in single-digit milliseconds. Ranking: a wide-and-deep or DCN model scores the 500 candidates with dense features (video age, past watches, device) and cross features; latency budget ~60 ms. Re-ranking: diversity rules, freshness boosts, policy filters trim to <50 items.

Problem: Serve a 70B–175B parameter autoregressive language model to millions of concurrent users with <500 ms time-to-first-token and >20 tokens/sec per user, while keeping cost per million tokens economically viable.

Scale: Tens of thousands of H100/A100 GPUs, peak load of ~100k concurrent inference requests, output sequences of 100–4000 tokens.

Architecture: Each model replica is tensor-parallel (TP=8, one node); replicas serve traffic behind a load balancer. The serving engine (vLLM-style) runs an iteration-level scheduler rather than request-level.

Problem: For a query typed by a user, retrieve and rank billions of web documents to produce a top-10 result page in ~200 ms — with freshness (breaking news must surface within minutes), quality (spam, low-quality pages must be demoted), and diversity (multiple result types: web, video, image, knowledge panel).

Scale: Billions of documents, billions of queries per day, petabytes of crawl data refreshed continuously.

Architecture: A multi-tier retrieve-and-rank pipeline. Indexing layer: Distributed inverted index (Bigtable/Colossus) stores term → document posting lists. Match/recall layer: BM25 or equivalent fast text match produces ~1000 candidates per query shard. Learning-to-rank (LTR) layer: a gradient-boosted tree or neural ranker scores the candidates on hundreds of features (PageRank, query-doc relevance signals, freshness, click feedback). Result blender: merges ranked lists from vertical indexes (images, videos) into a single page.

Problem: Present each of 230 M+ subscribers with a personalized homepage — rows of titles (Continue Watching, Top Picks, Trending) and optimized artwork for each title — that maximizes long-term engagement (hours watched per month) rather than just immediate click-through.

Scale: 230 M subscribers, ~15k titles, but the key challenge is heterogeneous signals: a user watches 2–3 titles per week (sparse labels), yet Netflix must personalize across dozens of row types and artwork variants.

After four very different case studies, the same five structural patterns recur. Knowing these lets you reverse-engineer any new system quickly.

The takeaway: when you are asked to design any ML system from scratch, quickly ask yourself about each of these five axes. Silence on any one of them is a red flag to interviewers.

What it is: Standard transformer self-attention is $O(n^2)$ in sequence length $n$ for both compute and memory. A 128k-token context with a 7B model requires tens of gigabytes of KV cache per sequence, and the attention computation itself dominates. This makes long-context generation extremely expensive.

Why it's unsolved: Subquadratic attention methods (linear attention, state-space models like Mamba, sliding-window attention) trade off quality for efficiency, and no method cleanly dominates standard attention on all tasks. KV cache compression (quantizing KV entries, evicting less-attended tokens) introduces approximation error that is hard to bound theoretically. The problem interacts with hardware: KV memory is a bottleneck at 128k context but may be a non-issue for 1k contexts, so no universal solution exists.

What a strong candidate says: "For long context today, I'd use GQA/MQA to reduce KV heads, quantize the KV cache to int8, and implement eviction of low-attention tokens using something like StreamingLLM. For the compute side, FlashAttention-3 with kernel-level IO minimization. I'd design my system to serve short and long contexts on different hardware pools because the bottleneck flips. Longer-term I'm watching hybrid architectures that combine a sliding-window attention with full attention at sparse positions."

What it is: ML systems today are overwhelmingly trained in discrete batch retrains: collect data → train → deploy → repeat on a schedule (daily, weekly). The ideal is a model that updates continuously from new data, like a human learning from experience. "Continual learning" or "online learning" describes this goal.

Why it's unsolved: Three hard sub-problems resist easy solution simultaneously. Catastrophic forgetting: a model finetuned on new data tends to overwrite what it learned earlier. Distribution shift under the model's own influence: once the model changes what users see, the data distribution changes — the model is training on its own outputs, a feedback loop that can spiral. Infrastructure complexity: online training requires the training and serving codepaths to be deeply integrated, with production-safe rollback when a new checkpoint degrades quality. None of these is theoretically solved; industrial practice uses frequent-but-not-continuous retrains (hourly at the extreme end for news feeds) as a pragmatic middle ground.

What a strong candidate says: "I'd be cautious about true online learning in production. The safer path is shortening the retrain cadence — move from weekly to daily to hourly batch retrains, with automated quality gates before each promotion. For the truly time-sensitive signal, I'd use a lightweight online component (e.g., a bias correction layer or a user-specific bandit layer) that updates in real time while the main model stays on a daily schedule. This decouples the risk."

What it is: Traditional retrieval-ranking pipelines are a two-step process: first retrieve candidates with a fast approximate system (ANN over dense embeddings or BM25), then rank candidates with an expensive model. Generative retrieval proposes to collapse these into one: a language model directly generates the document identifier (e.g., a string ID, a URL, a semantic code) for the relevant document, bypassing the index and the retrieval stage entirely.

Why it's unsolved: Scaling is the core difficulty. A language model must memorize all document identifiers in its parameters — for a web-scale corpus of billions of documents, this is infeasible with current architectures. Furthermore, new documents require retraining (or at minimum, careful finetuning) of the entire model, not just an index update. Incremental indexing — a solved problem for classical retrieval — becomes a major open challenge. Early results on small corpora (~100k documents) are promising but have not translated to web scale. Hybrid approaches (a generative step to produce a "docid" then a lookup) partially bridge the gap but reintroduce the two-step structure.

What a strong candidate says: "Generative retrieval is exciting for the insight that retrieval and ranking objectives can be unified. Today I'd still use a dense retrieval index for anything over a few million documents. The place where generative retrieval is already practical is entity lookup — asking a model to generate the canonical name of an entity in a structured KB — because the ID space is small and stable. I'd watch this space for the next 2–3 years."

What it is: A modern AI application is often a compound system: a chain of LLM calls, retrievers, rerankers, formatters, and tool calls, assembled into a pipeline. Each component has parameters (prompts, which model to use, what to retrieve, how many results). Optimizing the whole pipeline jointly — rather than each component in isolation — is the problem that DSPy and similar frameworks attempt to solve.

Why it's unsolved: The pipeline is non-differentiable end-to-end: discrete decisions (which documents to retrieve, what prompt template to use) break gradient flow. The search space over joint prompt-configuration-model-selection is combinatorially vast. And the evaluation signal is often delayed, noisy, or expensive (human preference). Current DSPy-style approaches use discrete optimization (few-shot bootstrap, greedy instruction search) that work well for small pipelines but scale poorly — a 10-component pipeline with 5 choices per component has 5¹⁰ ≈ 10M configurations to explore. LLM-based optimization meta-prompts reduce this somewhat but introduce their own instability.

What a strong candidate says: "For compound system optimization today, I'd use a combination of: (1) modular evaluation — measure each stage's contribution independently so I know where the bottleneck is; (2) prompt optimization tools like DSPy or OPRO for the discrete prompt parameters; (3) model selection per stage based on cost/quality curves; and (4) end-to-end eval on held-out cases to catch emergent failures. I'd resist the urge to jointly optimize everything — the search space is too large and overfitting to the eval set is a real risk."

What it is: OpenAI o1 and similar systems discovered that allowing a model to "think longer" at inference time — generating an extended chain of thought before answering — dramatically improves performance on hard reasoning tasks. This creates a new axis: instead of scaling model parameters, scale the number of inference tokens (and thus FLOPs) at serve time. The compute budget is now a first-class decision variable per request.

Why it's unsolved: Three interacting problems emerge. Budget allocation: how many tokens should a given request be allowed? Too few wastes quality; too many wastes cost. Good policies for dynamic budget allocation based on question difficulty are not yet well understood. Verification: "thinking longer" only helps if the model can recognize when it's found the right answer. A verifier model (process reward model, outcome reward model) is needed, but training such verifiers reliably is hard — they overfit to superficial patterns. Serving economics: a request that generates 8000 thinking tokens before a 50-token answer turns the cost model upside down (output tokens are 3–5× more expensive than input tokens in API pricing). Load prediction becomes very hard when per-request output length varies by 100×.

What a strong candidate says: "For serving systems that support extended thinking, I'd design for high variance in output length: use continuous batching, track per-request token budgets, and preempt requests that exceed their budget. On the cost side, the output-to-input token cost ratio (~3–5×) means that thinking-heavy workloads cost dramatically more per useful answer than non-thinking workloads — I'd price differentiate and expose a 'thinking budget' parameter in the API. For the allocation policy, I'd start with a simple heuristic (long questions get more tokens) and measure whether quality improves, before investing in a learned budget allocator."

1. Why does training-serving skew happen? Two implementations of one feature definition drift; fix with one definition, two materializations, logged-at-scoring features, parity tests.
2. Point-in-time correctness? Training joins must see feature values as of the label event time — never later — or the future leaks in and offline metrics lie.
3. Why a retrieval→ranking funnel? Can't afford the big model on 100M items in 100ms; cheap recall first, expensive precision on hundreds.
4. Why dot-product retrieval? It's the only scoring function ANN indexes can search in sublinear time — model expressiveness traded for searchability.
5. In-batch negatives + logQ? Other examples' positives serve as free negatives; subtract log-popularity so frequent items aren't unfairly punished.
6. Why calibrate ranking scores? Scores get added in value formulas and thresholded — operations that need real probabilities, not just correct order.
7. Position bias one-liner? Clicks confound quality with exposure; train with position then freeze it at serving, or reweight by examination propensity.
8. p50 vs p99 — why obsess over the tail? Fan-out: a page touching 50 services hits a slow one with probability 1−0.99⁵⁰ ≈ 40%; the tail IS the user experience.
9. Why dynamic batching? GPU does 1 inference in 10ms and 32 in 12ms; amortize or run at 5% utilization and 10× cost.
10. Little's law cameo? Concurrency = arrival rate × latency; it sizes worker pools and exposes where queueing time hides.
11. Adam memory rule? ~16 bytes/param in mixed precision (bf16 weight+grad, fp32 master+two moments) — 7B params ≈ 112GB before activations.
12. ZeRO stages? Shard, in order: optimizer states (1), +gradients (2), +parameters (3/FSDP) — climbing only as far as the byte math forces.
13. Why does TP stay in-node? Per-layer collectives need NVLink bandwidth; cross-node TP drowns in an ~18× bandwidth cliff.
14. Pipeline bubble? (p−1)/(m+p−1) idle fraction — more microbatches m amortize the fill/drain cost.
15. Why checkpointing is non-negotiable? 10k GPUs × 1 failure/3yr each ≈ one failure every 2.6 hours; cadence from T ≈ √(2·write/λ).
16. Grad-norm spike at 3am — first move? Triage order: cluster health → gradient norms → the data batch → LR schedule → precision/loss-scale → rollback+skip.
17. Prefill vs decode? Prefill processes the prompt in parallel (compute-bound); decode emits one token at a time streaming all weights (bandwidth-bound). Same model, two physics.
18. KV cache in one breath? Cache each token's K,V per layer so token t does O(t) attention instead of recomputing O(t²)-ish history every step; memory = 2·layers·kv_heads·head_dim·bytes·len.
19. Why continuous batching? Sequences finish at different times; refill slots per-iteration instead of idling until the longest finishes — 2-10× throughput.
20. Paged attention? Virtual memory for KV: on-demand blocks + block table kill the reserve-max-length fragmentation that capped batch size.
21. Speculative decoding guarantee? Draft proposes, target verifies with accept/reject math — output distribution exactly the target's; speed without quality risk.
22. Why are output tokens pricier than input? Input is one parallel cacheable prefill pass; output is serial bandwidth-bound decode — more GPU-seconds per token.
23. RAG quality is bad — first probe? Measure retrieval recall@k on labeled queries FIRST; generation can't cite what retrieval never fetched. Binary-search the pipeline.
24. Why is RLHF infra hard? Four models live at once and generation (an inference workload) sits inside the training loop — rollout throughput gates everything.
25. LoRA's serving superpower? Hundreds of tenants share one frozen base; adapters are tens of MB, hot-swappable, batchable.
26. Drift taxonomy? Data drift P(X), concept drift P(Y|X), label shift P(Y) — different alarms, different fixes; PSI > 0.2 = act.
27. Why eval gates? "Reward improved" is a proxy; the gate catches capability regressions, safety failures (both directions), and format breaks before users do.
28. Capacity recipe? Demand → unit work → unit capacity × utilization haircut (30-50% MFU is good) → divide → sanity-check against a known system.
29. Launch ladder? Offline → shadow → canary → A/B → 100% + holdback; each rung catches what the previous structurally cannot.
30. The 2am playbook? Deploy? → pipeline lag? → feature nulls? → score distribution? → segment breakdown? → upstream product change? Cheapest-first, always.