Domain / Project Deep Dive

"The ranker was multi-task, so the loss was a weighted sum of per-task losses. Click, dwell-binary, and reaction were binary cross-entropy; dwell-time-regression was a Huber loss because it's robust to the long tail of session times that an L2 would let dominate. The task weights weren't tuned by hand — I treat manual loss weights as a smell — we used uncertainty weighting so each task's weight scales inversely with its learned noise, which stopped the high-variance dwell head from drowning the click head. For retrieval, the two-tower used a sampled-softmax loss with a log-Q correction to debias the in-batch negatives toward their true sampling frequency, because popular items appear as negatives disproportionately and without the correction the model under-ranks popular content."

"Offline I used recall@k for retrieval and per-head AUC plus calibration error for ranking, all on a temporally held-out set — never a random split, because a random split leaks future information into training and inflates the metric. Online was the source of truth: A/B with session length as primary and the integrity guardrails. The offline-online gap is expected and informative: I track the correlation between offline metric deltas and online metric deltas over many launches. A launch with a strong offline win and a null online result is a leakage or skew flag — which is exactly how I caught the training-serving skew in the main story. The point isn't to make offline predict online perfectly; it's to know your offline metric's reliability and treat divergence as signal, not noise."

"On the platform project, the retry amplification that turned a single hot key into a platform-wide cascade was my gap — I'd designed the cache for the happy path and the retry policy was naive: fixed retries, no backoff, no coalescing. I owned it in the postmortem without hedging, and the fix I wrote — request coalescing plus backoff with a retry budget — became the default for new caching layers in the org. The deeper lesson I took: at platform scale, the failure mode you didn't design for is the one that pages you, so I now design the degradation path before the happy path." Note: name a real mistake, own it cleanly, and end on the systemic fix and the durable lesson. Never pick a fake-humble "I worked too hard" mistake — it reads as evasion at this level.

"Two things. First, retrieval: I'd seriously evaluate a generative-retrieval or learned-index approach instead of two-tower-plus-ANN — encoding the catalog into a model that directly generates candidate IDs collapses the recall ceiling problem we monitored. I'd prototype it against our two-tower baseline before committing, because the operational maturity of ANN is real and I wouldn't trade it cheaply. Second, the calibration and skew monitoring I bolted on after the fact should have been in the platform from day one — if I rebuilt the platform, point-in-time-correct features and a skew monitor would be non-optional infrastructure, not a per-team add-on. The willingness to say 'two-tower might not be what I'd choose now' matters — staying current and being honest that your past choice has aged is a senior signal."

"On the staged-migration call, a senior engineer strongly wanted a full rewrite — cleaner end state, less shim code. They had a real point. I disagreed because the staging let us attribute impact per stage and roll back independently. Rather than overrule, I framed it as a risk question: what's our recovery story if the new ranker regresses? We didn't have a good one for the big-bang approach. I proposed we stage retrieval first as a test of both the approach and the hypothesis, and agreed that if retrieval went cleanly we'd revisit collapsing the later stages. The data made the call — and when ranking did regress, the staging is what saved us, which retroactively settled the debate. The principle: disagree on the decision, align on the criteria, let the criteria decide."

"This is the question I'd want asked. The team of nine built the funnel — retrieval training, the MMoE-to-PLE migration, the serving integration. What I personally designed and own: the calibration layer architecture, the staged-migration plan and the per-stage holdout design, and the training-serving skew root-cause — I wrote the feature-logging instrumentation and did the feature-by-feature diff that found the occurrence-time leak. I also designed the skew monitor that became an org default. As tech lead I set the technical direction across the three teams, but I'm careful not to claim the team's implementation as mine — the leveling signal is owning the hardest 20% precisely, not claiming 100%."

"Graph-based recall — using the follow graph and engagement graph — was actually part of the candidate sourcing; two-tower complemented it rather than replacing it, because the graph is great for known connections and bad for cold or novel content, while the learned embeddings generalize across the long tail. A transformer over the full catalog at retrieval time is a latency non-starter at 50M items and an 80ms budget. The architecture is constraint-driven: latency budget rules out cross-attention at recall, catalog size rules out exhaustive scoring, and the cold-start need rules out graph-only. Two-tower plus ANN, complemented by graph sources, is what's left standing."

"Three mechanisms. The log-Q correction in retrieval debiases against popularity at training time. We logged the propensity — the probability the system had of showing each item — and used inverse-propensity weighting on the conversion labels so we weren't just learning 'what we already showed got engaged with.' And the reranker's diversity penalty actively counteracts collapse toward a few popular clusters. The feedback loop is the central failure mode of any deployed ranker, so I treated debiasing as a design requirement, not a patch."

"Two different problems. New content: the item tower can embed an item from its content features before it has any engagement history, so two-tower gives us a reasonable cold-item embedding day one; we also had an explicit fresh-content source with an exploration boost so new items get impressions to gather signal, with the boost decaying as engagement data accumulates. New users: the user tower leans on whatever context we have — declared interests, early session behavior — and we fall back to a popularity-and-diversity-tilted ranking until the personalization signal is strong enough. The judgment call was the exploration budget: too much and you hurt the median user's feed to benefit cold items, so we capped exploration impressions as a small fraction of each feed and measured the cold-start lift against the exploration cost."

"This is a product decision dressed as a modeling one, so I didn't let it be set by a single engineer's intuition. The final ranking score is a weighted combination of calibrated head probabilities — click, dwell, reaction, downvote-as-a-penalty. We bootstrapped the weights from a logistic fit against a north-star session-quality label, then treated them as tunable and A/B-tested weight changes the same as model changes. The downvote weight in particular was set conservatively because the integrity guardrail was non-negotiable — I'd rather leave a little engagement on the table than ship a feed that's optimized into rage-bait. Calibration is what makes these weights meaningful at all; uncalibrated heads make the weights uninterpretable."

"I set the stopping criterion up front, against the charter: a confirmed, sustained session-length lift with guardrails neutral. Once retrieval, ranking, and reranking had each shipped and the funnel-wide holdout held its win over a full four-week ramp including a weekend cycle, the project's mandate was met. Continuing past that would have been diminishing returns chasing fractions of a percent — better to hand the now-stable platform to the team and redirect my time to the next-largest lever. Knowing when to stop and reallocate is a leverage decision, not a finishing-line decision."

"I'd protect retrieval recall and the calibration layer because they're cheap and high-leverage, and I'd cut ranker capacity first — distill the ranker to a smaller student, since we'd already shown a distilled model held 99% of offline AUC at 40% of the FLOPs. If that still didn't fit, I'd reduce the candidate count from 1,000 to maybe 600 and accept a small recall ceiling drop, which I'd monitor with recall@k. The principle is to cut where the quality-per-millisecond curve is flattest, and we'd measured that curve precisely so the decision is data-driven, not a guess."

"Freshness is per-feature, set by a sensitivity analysis, not a blanket policy. Session-recency features were online with sub-second freshness because they're the highest-signal and stalest-sensitive. Counting features — interaction counts over windows — were batch with hourly-to-daily freshness because their value barely moves intra-day, and computing them online would have tripled feature-store cost for negligible quality. The model itself we retrained daily with a warm start and full retrains weekly; the binding freshness constraint was never the model, it was the features, which is the non-obvious thing I'd want a junior engineer to internalize."

"That's the interference problem — the treatment model's serving pollutes the shared training data. Two mitigations. We held the treatment and control on separate, fixed user buckets so a user always saw one variant, avoiding within-user contamination. And for the training data, we logged which variant generated each impression and trained on variant-consistent data during the experiment so the control model wasn't learning from treatment-generated logs. For network effects — content popularity is shared across buckets — we accepted some leakage and sized the experiment large enough that the bias was small relative to the effect, and we cross-checked with a separate held-out cluster where feasible. Being explicit about interference is itself a senior signal; most people forget the training data is shared."

"Separate single-task models would have been cleaner per task but tripled serving cost — one forward pass per task per candidate over 1,000 candidates blows the latency budget. The whole point of the multi-task architecture is to share the expensive lower layers across tasks and pay for the heads only. The reason it's MMoE and not shared-bottom is negative transfer: shared-bottom forces one representation, and our click and dwell objectives partly conflict, so improving one hurt the other. MMoE gives each task a learned gate over shared experts, which lets the tasks route to different expert combinations. We moved to PLE when even MMoE seesawed, because PLE adds task-specific experts alongside the shared ones — it's the standard progression when you see task interference, and I can sketch the gating math if useful."

"I didn't assume skew — I ranked hypotheses by likelihood and cost-to-test. The candidates were: (1) the offline metric doesn't correlate with the online metric for this change, (2) a guardrail or product surface difference between experiment and offline eval, (3) training-serving skew, (4) a bug in the experiment setup like bucket contamination. I ruled out (4) first because it's cheap to check — verified bucket assignment and traffic split. I checked (1) by looking at historical offline-online correlation for similar launches, which was usually high, making it unlikely. That left skew and product-surface differences, and the feature-logging diff is what isolated skew specifically. The meta-point: I treat a confusing result as a hypothesis-ranking problem, cheapest-and-most-likely first, not as a single guess to confirm."

"Three layers. Operational: p99 latency, error rate, GPU utilization, cache hit rate, with auto-rollback triggers. Model-health: per-head calibration error and AUC on a streaming holdout, prediction-distribution drift, and the feature-skew monitor comparing serving versus training feature distributions. Business: the primary and guardrail metrics on a dashboard with anomaly alerts. The skew monitor is the one I'm proudest of because it's predictive — it catches the cause of a regression before the business metric shows the symptom. Monitoring the model's inputs, not just its outputs, is the difference between finding out from a dashboard and finding out from a root-cause investigation three weeks later."

"It's a flexibility-versus-data tradeoff. Platt scaling fits a single sigmoid — two parameters — which assumes the miscalibration has a specific logistic shape; temperature scaling is even more constrained, one parameter that just sharpens or softens. Those are great when you have little calibration data because they can't overfit, but they can't correct a non-monotonic-in-shape or oddly-shaped miscalibration. Isotonic regression fits an arbitrary monotonic non-decreasing map, so it can correct any shape of miscalibration as long as it preserves rank order — which is exactly what I need, since the value model only cares that calibrated scores are comparable, and monotonicity guarantees I never reorder within a head. The cost is that isotonic can overfit on small samples, so I fit it on a large daily held-out log and refit daily. If I'd had sparse calibration data per head I'd have fallen back to Platt; with abundant logged data, isotonic's flexibility wins. The judgment is matching the calibrator's capacity to the calibration data you actually have."

Be ready to put this on the board cleanly. "The final score for candidate $i$ given user $u$ is a weighted sum of calibrated head probabilities:"

"$\hat p_h$ is the raw model probability for head $h$ — click, dwell, reaction. $g_h$ is that head's isotonic calibration map, so $g_h(\hat p_h)$ is an actual probability comparable across heads — without it the weights are meaningless. $w_h$ are the value-model weights, bootstrapped from a logistic fit to a session-quality north-star label and then A/B-tested like any model change, because they're really a product decision. The downvote head enters with a negative sign and a conservatively-set $w_{\text{dv}}$ — I'd leave engagement on the table before shipping a feed optimized into rage-bait, because the integrity guardrail is non-negotiable. The reranker then applies diversity and policy on top of this score over the top ~50. Every term has a reason: calibration makes the weights meaningful, the weights encode the product tradeoff, the negative downvote term encodes the guardrail."

"It connects directly, and it's worth being precise that those are deliberately not learned rankers — they're principled heuristics, and that's a feature. Reddit's hot score is roughly $\text{sign}(u-d)\cdot\log_{10}\max(|u-d|,1) + t/45000$ — a log-dampened vote signal plus a linear time term giving about a 12.5-hour half-life, so it self-demotes old content without a model. Comments use the lower bound of a Wilson confidence interval on the up/down ratio, which is exactly the right tool: it ranks by a statistically defensible estimate of quality given small sample sizes, so a comment with 2 upvotes doesn't out-rank one with 200 just because the ratio is higher. The connection to my work: I'd treat those as the strong, interpretable baseline and the cheap candidate-scoring layer, and reserve the learned multi-task ranker for where personalization actually beats the heuristic. The judgment is knowing when a well-chosen heuristic is the right answer — a Wilson lower bound or a log-decayed score is cheap, debuggable, and unbiased, and you only pay for a learned model where it earns its keep. Over-engineering a transformer where a Wilson score suffices is an anti-pattern, not a strength."

"They're the same funnel shape, which is reassuring — it means the architecture generalizes. Reddit's notification system is reportedly budgeting, then retrieval, then ranking, then rerank: a causal budgeting stage decides how many notifications a user should get (so you don't over-notify), a two-tower retrieval stage narrows candidates, a multi-task DNN ranks them, and a business-logic rerank applies policy. My home-feed funnel is candidate sourcing → ranking → calibration → rerank — structurally identical except notifications add an explicit budgeting stage up front because the cost of a bad notification is much higher than a bad feed item; a mistimed push annoys and churns a user, so you gate volume causally first. If I were adapting my funnel to notifications, the budgeting stage is the piece I'd add and I'd lean hard on the integrity-guardrail discipline, because 'remember the human' is most at stake when you're interrupting someone outside the app."

"Causal attribution is the whole game and I treat it skeptically about my own work. The primary instrument was a randomized A/B with users in fixed, mutually-exclusive buckets — random assignment is what licenses the causal claim, so I guard the randomization carefully and check for bucket imbalance on pre-period covariates before trusting any result. I ran it over a full multi-week cycle including a weekend, because day-of-week and novelty effects can masquerade as a sustained lift in a short window. For attribution within the funnel, the staged-migration design was deliberately causal: each stage shipped behind its own holdout, so I can say retrieval contributed ~60% of the +3.1% rather than attributing the whole funnel to one change. When a launch was confounded — say a concurrent infra change landed in the same window — I'm explicit that I can only report the offline delta plus a partial online read, rather than claiming a clean causal number. The tell of someone who actually owned impact is that they volunteer the confounds; the tell of someone who didn't is a suspiciously clean story."

"That's the proxy-validity problem, and it's where ranking work gets genuinely hard. Short-term engagement proxies are cheap and fast but can diverge from — or even oppose — long-term retention; the classic failure is optimizing a proxy like click-through and degrading the thing you actually care about. My approach: treat the proxy as a hypothesis about the long-term goal and periodically validate it. We ran longer holdouts — multi-month back-tests where a small holdout never saw the new model — specifically to read retention, which a two-week A/B can't. When the short-term proxy and the long-term metric disagreed, I trusted the long-term one and treated the divergence as evidence the proxy needed fixing — that's actually how the quality-metric work in Story 3 started, when engagement was up but the long-term 'worth your time' signal was flat. The principal move is refusing to declare victory on a proxy you haven't validated against the real goal."

"The hot-key cascade is the cleanest example. Detection: the p99-latency auto-alert fired within ~60 seconds — I monitor inputs and operational metrics, not just business metrics, precisely so detection is fast. Triage: on-call's first job was stop the bleeding, not diagnose, so they hit the global kill switch that reverts serving to last-known-good in one command; that stabilized in about two minutes. Only then did we diagnose, which is the right order — mitigate first, root-cause second. Root cause in the postmortem: three contributing factors — no hot-key detection, unbounded retries with no backoff, no request coalescing — and I was explicit that the retry amplification was my design gap. Prevention, which is the part that levels you: request coalescing so concurrent misses for one key share a single computation, exponential backoff with jitter and a retry budget, and a hot-key detector that replicates a saturating key. I wrote the coalescing layer myself and it became the default for new caching layers. The shape I'd want a junior to learn: detect on inputs, mitigate before diagnosing, blameless postmortem, and fix the class of bug, not the instance."

"Both are forms of the system learning from data it generated. Position bias: items shown higher get clicked more regardless of relevance, so naive training conflates position with quality. We modeled position explicitly — feeding position as a feature during training so the model can factor out its effect, and dropping it (or fixing it to a constant) at inference so we score on relevance, not position. Selection bias is deeper: we only have labels for items the system chose to show, so the model never learns about the candidates it never surfaced. We mitigated with inverse-propensity weighting — logging the probability the system had of showing each item and weighting labels by the inverse — plus an explicit exploration budget that shows some under-served candidates to gather unbiased signal. Neither fully solves it; the honest framing is that a deployed ranker trains on its own choices, so debiasing is a permanent design requirement, not a one-time fix."

"Yes, and I'd want you to ask. Here's the test: the calibration layer, the staged-migration-and-holdout design, the skew root-cause and the monitor that came out of it — those are things where if you removed me, they don't happen or happen much later, and my teammates would say the same because we did design reviews where those were visibly my proposals. Conversely, the MMoE-to-PLE migration and the serving integration were genuinely the team's execution — I set direction but a strong engineer drove each, and I'd be misrepresenting it to claim them. The reason I'm careful about this split isn't modesty, it's that over-claiming is the fastest way to lose credibility in a debrief: if my story doesn't survive a back-channel, nothing else I said matters. Precise attribution is a trust signal, and it's also just how you should think about leverage — my job as the senior person was to own the parts only I could and make the rest ownable by others."

"I treat it as a leverage question and I'm biased toward data and system fixes because they tend to compound, where model tweaks often don't. Concretely, I look at where the error is coming from. If offline metrics are strong but online is weak, that's a data or system problem — skew, leakage, freshness — and a better model won't help; that's exactly the home-feed skew story, where retraining a skewed model faster just learns the skew faster. If both offline and online are plateaued and the data is clean, then it's a modeling ceiling and architecture work — like the sequence model — is the lever. And if quality is fine but cost or latency is the constraint, it's a systems problem — distillation, batching, caching. The mistake I see juniors make is reaching for a fancier model when the real issue is a stale feature or a leaky join. Diagnose which of the three is binding before you invest, because effort in the non-binding one is wasted."

"This is the slice problem and it's the same shape as the cold-start skew in the real-time project — aggregate metrics hide localized failures. Finding it: I don't trust a single headline number; I monitor metrics sliced by the dimensions where harm is plausible — new vs established users, content language, creator size, traffic source — and alert on slice divergence, not just the mean. Fixing it depends on the cause: if it's a data-volume problem (the subpopulation is under-represented in training), I'd re-weight or up-sample, or add an exploration boost to gather signal; if it's a feature problem (a signal that's predictive for the majority is absent or noisy for the slice), I'd add slice-robust features or a fallback. The judgment call is that you sometimes accept a small aggregate cost to remove a slice harm, and at a place that cares about 'remember the human,' I'd argue that's the right trade and I'd want it measured, not hand-waved."

"Silent degradation has three usual causes — data drift, the world changing, and feedback loops — and each needs a different sensor. For drift, the feature-skew monitor compares live feature distributions to the training distribution on a schedule and alerts before the business metric moves, which is the point: input monitoring is leading, output monitoring is lagging. For the world changing, I watch per-head calibration error on a streaming holdout, because a model that was calibrated at launch drifts as behavior shifts, and miscalibration silently corrupts the value-model ordering. For feedback loops, I periodically compare the model's behavior on the exploration slice — which is closer to unbiased — against the exploited traffic, to catch the ranker collapsing into its own preferences. And we retrain on a cadence with warm starts. The principle: monitor the inputs and the calibration, not just the output metric, so you find out from a sensor, not from a quarter-end review."

"Because magnitude is mostly a function of surface area and baseline, not skill — a 0.5% lift on a mature, heavily-optimized billion-user surface can be a far harder and more valuable win than a 20% lift on a greenfield feature with a weak baseline. What should count is the difficulty of the problem, the rigor of the measurement, and what outlived the work. I'd rather defend a modest number I measured cleanly and a durable artifact — the skew monitor that protects models I never touched — than a flashy number I can't attribute. If anything, an outsized number on a mature system should invite skepticism about the measurement. So I'd point you at the method and the durability, not ask you to weigh my percentage against someone else's."

"I under-invested in the validation gate early. On the home-feed project I caught the skew by hand because the gate didn't exist yet; on the real-time project the gate existed but only checked aggregates, so it passed a slice-localized skew and I caught that one by hand too. The pattern I was slow to see: I kept building skew detection reactively, after each incident, instead of treating 'how do I know training matches serving' as a first-class platform requirement from day one. The deeper fix wasn't a better gate, it was recognizing that for any team deploying learned features, train-serve consistency validation should be infrastructure you inherit, not something each project reinvents after getting burned. I eventually pushed for that as a platform default, but I'd have saved two incidents if I'd seen it as foundational the first time rather than the third."

"I set kill criteria up front, which is the discipline that makes stopping possible — you can't make a clean stop decision in the middle of sunk cost without pre-committed thresholds. On the home-feed project the early kill gate was retrieval: if learned two-tower didn't beat heuristic recall offline by a clear margin in the first couple of months, the project's premise was wrong and I wanted to find that out in month two, not month twelve, so retrieval was sequenced first precisely as a cheap test of the thesis. On the real-time project the kill gate was the cost-versus-lift threshold I committed to when getting the infra approved. The principal habit is to design the project so the riskiest assumption is tested first and cheapest, with a pre-agreed threshold, so that stopping is a planned outcome rather than an admission of failure. Knowing when to stop and reallocate is leverage, and it's a lot easier to exercise when you decided the criteria before you were emotionally invested."

"This is a nice test of whether the architecture I described is actually extensible, and it is by design. First I'd ask whether it belongs in the learned value model or the rule-based reranker — and creator diversity is a policy goal that shifts over time, so I'd start it in the reranker, not bake it into a head, exactly like the other diversity and integrity policy lives there. That gives a fast, retrain-free path to ship and tune it. If we found the reranker was too blunt — say it was demoting good content from large creators rather than promoting good content from small ones — then I'd promote it to a learned signal: add a creator-size-aware head or feature so the model trades it off smoothly. The sequencing principle is ship the cheap reversible version in policy first, learn whether it works, and only pay the retrain cost to make it learned once you've validated the objective is worth encoding. And I'd insist on a metric for 'creator diversity' before touching anything — same discipline as the quality work: don't optimize a goal you haven't made measurable."

"I distinguish debt I take on knowingly from debt I take on accidentally. The staged migration deliberately took on debt — the shim code, two systems live at once — because the alternative, a big-bang rewrite, traded that debt for unbounded rollback risk, and I'd rather carry shim debt I can pay down on a schedule than risk a regression I can't isolate. So I made the debt explicit: every shim had a deprecation date and an owner, and the migration wasn't 'done' until the debt was retired. Accidental debt — the kind that accretes from rushing — I guard against with the validation gates and monitoring, because the most expensive debt in ML systems isn't ugly code, it's an undetected skew or a silently degrading model. The judgment is which debt compounds. Code debt is usually linear and payable; data and consistency debt compounds and pages you, so I spend my rigor there and let the code debt be pragmatic."