All topics
Interview Preparation
ML System Design
End-to-end design of recommender, fraud, search-ranking, CTR, moderation, dedup, churn, chatbot, matcher and ETA systems, from metric to serving cost.
Sort:
Level:
Type:
14 questions
SeniorDesignVery commonYou are designing click-through-rate prediction for an ad exchange. Scoring runs at about 100k queries per second with a p99 budget under 20 ms, over billions of impressions per day. The bid for each auction is computed as an expected value — the predicted click probability multiplied by the advertiser value — so the number the model emits is spent directly. Features are dominated by high-cardinality sparse categoricals such as ad id, publisher and device. Walk through how you handle those features, which model you serve and why, what calibration means here, and how often you retrain.
SeniorDesignVery common
You are designing click-through-rate prediction for an ad exchange. Scoring runs at about 100k queries per second with a p99 budget under 20 ms, over billions of impressions per day. The bid for each auction is computed as an expected value — the predicted click probability multiplied by the advertiser value — so the number the model emits is spent directly. Features are dominated by high-cardinality sparse categoricals such as ad id, publisher and device. Walk through how you handle those features, which model you serve and why, what calibration means here, and how often you retrain.
High-cardinality categoricals go through hashing or embeddings. A 20 ms budget at 100k QPS favours a shallow model over a deep ensemble. Because the bid multiplies the probability, it must be calibrated, not merely well-ranked, and retrained daily.
Common mistakes
- ✗Assuming good ranking implies good probabilities when the score is multiplied into a bid
- ✗One-hot encoding high-cardinality categoricals instead of hashing or embedding them
- ✗Retraining on a slow cadence while the ad distribution shifts within days
Follow-up questions
- →Which calibration method — say isotonic regression, a monotonic post-fit — would you use here?
- →How would you detect calibration drift in production before the revenue numbers move?
SeniorDesignVery commonYou are designing real-time antifraud for card payments. The stream runs at about 2000 transactions per second and a decision must return in under 100 ms. Confirmed fraud is roughly 0.1% of volume, and chargeback labels arrive 30 to 60 days after the transaction and are themselves noisy. The fraudsters adapt to whatever you deploy. A human review queue exists, but analysts can only work a limited number of cases per day. Walk through the evaluation metric you would defend, how you train and validate with delayed noisy labels, how you pick the decision threshold, and what you monitor after launch.
SeniorDesignVery common
You are designing real-time antifraud for card payments. The stream runs at about 2000 transactions per second and a decision must return in under 100 ms. Confirmed fraud is roughly 0.1% of volume, and chargeback labels arrive 30 to 60 days after the transaction and are themselves noisy. The fraudsters adapt to whatever you deploy. A human review queue exists, but analysts can only work a limited number of cases per day. Walk through the evaluation metric you would defend, how you train and validate with delayed noisy labels, how you pick the decision threshold, and what you monitor after launch.
At a 0.1% fraud rate accuracy and ROC-AUC flatter a model catching nothing, so judge precision and recall at the operating point via PR-AUC. Train on matured 30-60 day labels, set the threshold from review capacity and decline cost, and watch drift.
Common mistakes
- ✗Reading high accuracy or ROC-AUC as success when positives are 0.1% of volume
- ✗Treating late chargeback labels as if they were available at scoring time
- ✗Picking the threshold from the model alone, ignoring review capacity and decline cost
Follow-up questions
- →How would you notice an adversary has adapted before the chargeback labels confirm it?
- →How do you price a false decline against a missed fraud when setting the operating point?
JuniorTheoryCommonWhat are the standard stages of designing an ML system end to end?
JuniorTheoryCommon
What are the standard stages of designing an ML system end to end?
Frame the business problem and the decision the model drives, then fix offline and online metrics. Secure data and labels, engineer features, pick a baseline before a complex model, serve within the latency budget, and monitor with retraining.
Common mistakes
- ✗Jumping to model choice before the business decision and the metric are fixed
- ✗Treating serving latency and monitoring as post-launch infrastructure work
- ✗Skipping the simple baseline that the complex model is supposed to beat
Follow-up questions
- →Why fix the online metric before the offline one, and how do you link them?
- →What does a baseline give you beyond a number the model must beat?
MiddleDesignCommonA subscription product has about 3M active subscribers on monthly billing. The retention team can personally reach only about 5000 users a week, and the model score triggers a discount offer. Walk through the design and spend most of it on the label — how you define the churn horizon on monthly billing, what you do with users whose horizon has not yet elapsed, where you put the feature cutoff so that post-cutoff events cannot leak into training, and how the retention team should actually use the score given that it can contact well under one percent of the base each week.
MiddleDesignCommon
A subscription product has about 3M active subscribers on monthly billing. The retention team can personally reach only about 5000 users a week, and the model score triggers a discount offer. Walk through the design and spend most of it on the label — how you define the churn horizon on monthly billing, what you do with users whose horizon has not yet elapsed, where you put the feature cutoff so that post-cutoff events cannot leak into training, and how the retention team should actually use the score given that it can contact well under one percent of the base each week.
Churn is no renewal within a horizon after billing; unelapsed horizons are censored, not called retained. Features stop at the observation date, so later events cannot leak. Rank outreach by expected saved value.
Common mistakes
- ✗Labelling users whose churn horizon has not elapsed as retained
- ✗Building features from events that happened after the observation cutoff
- ✗Targeting outreach by raw churn probability instead of expected saved value
Follow-up questions
- →Which subscriber attributes are the likeliest to leak the churn label on monthly billing?
- →How does the discount offer itself corrupt the labels of the users you contacted?
MiddleDesignCommonA marketplace publishes about 10M user items a day, text plus images. Moderation is a small team — a few dozen analysts, each reviewing on the order of a few hundred items a day. The worst categories carry a legal SLA measured in minutes. Abusive items are well under 1% of the flow, and a brand-new abuse type can appear overnight and spread within hours. Walk through the design — how you order cheap and expensive checks, how you pick the auto-block and auto-allow thresholds and what happens to the grey zone between them, how analyst decisions become training labels, and what you ship on day one against an abuse type the model has never seen.
MiddleDesignCommon
A marketplace publishes about 10M user items a day, text plus images. Moderation is a small team — a few dozen analysts, each reviewing on the order of a few hundred items a day. The worst categories carry a legal SLA measured in minutes. Abusive items are well under 1% of the flow, and a brand-new abuse type can appear overnight and spread within hours. Walk through the design — how you order cheap and expensive checks, how you pick the auto-block and auto-allow thresholds and what happens to the grey zone between them, how analyst decisions become training labels, and what you ship on day one against an abuse type the model has never seen.
Cascade cheap rules, a light model, then the heavy multimodal model only on survivors. Auto-block and auto-allow thresholds bound a grey zone routed to analysts, whose verdicts become labels. New abuse gets rules, then a retrain.
Common mistakes
- ✗Cutting with one threshold instead of separate auto-block and auto-allow bounds
- ✗Running the expensive model over the whole flow rather than on cascade survivors
- ✗Waiting for a retrain to answer a new abuse type instead of shipping rules first
Follow-up questions
- →How do you place the auto-block threshold when a wrong block carries legal risk?
- →Analysts only ever label the grey zone — how does that bias the retrained model?
MiddleDesignCommonA product stakeholder walks up with exactly one sentence — personalize the homepage. You have one quarter of engineering time and an existing homepage that shows every visitor the same static blocks. There is no brief, no target number, and no named data owner. Before drawing a single box of an ML design, walk through the questions you must ask first — who the users are and what decision the system actually makes, the single success metric (an OEC, overall evaluation criterion) plus the guardrail metrics that protect it, what data and labels already exist today, the latency, cost, privacy and team constraints you must fit inside, and the baseline any model has to beat.
MiddleDesignCommon
A product stakeholder walks up with exactly one sentence — personalize the homepage. You have one quarter of engineering time and an existing homepage that shows every visitor the same static blocks. There is no brief, no target number, and no named data owner. Before drawing a single box of an ML design, walk through the questions you must ask first — who the users are and what decision the system actually makes, the single success metric (an OEC, overall evaluation criterion) plus the guardrail metrics that protect it, what data and labels already exist today, the latency, cost, privacy and team constraints you must fit inside, and the baseline any model has to beat.
Ask who the users are and what decision the system makes, then fix one success metric with guardrails. Check which data and labels exist today, the latency, cost, privacy and team limits, and the baseline the model must beat.
Common mistakes
- ✗Collecting many metrics instead of committing to one metric with guardrails
- ✗Starting from the model and the data pipeline before the user decision is named
- ✗Treating the existing non-personalized homepage as no baseline at all
Follow-up questions
- →How do you pick guardrail metrics that catch harm the success metric hides?
- →What would you ask if labels for the target decision do not exist yet?
SeniorDesignCommonYou are asked to build dynamic pricing for a marketplace with roughly 1M SKUs where prices may change once a day. Your historical data contains only the prices you actually charged, so you never observe what a different price would have earned. Buyers notice price movement, and visible volatility costs real trust — support already fields complaints about it. Walk through what you actually optimize and why a single-objective revenue maximizer is dangerous, how you obtain counterfactual data at all, how you would A/B test pricing without damaging trust, and what happens once your own prices become the training data for the next model.
SeniorDesignCommon
You are asked to build dynamic pricing for a marketplace with roughly 1M SKUs where prices may change once a day. Your historical data contains only the prices you actually charged, so you never observe what a different price would have earned. Buyers notice price movement, and visible volatility costs real trust — support already fields complaints about it. Walk through what you actually optimize and why a single-objective revenue maximizer is dangerous, how you obtain counterfactual data at all, how you would A/B test pricing without damaging trust, and what happens once your own prices become the training data for the next model.
Optimize revenue against retention guardrails — a pure revenue maximizer exploits buyers. Logs hold no counterfactual, so randomize prices on a small traffic slice. Test with a stable unit and price caps, since your prices become training data.
Common mistakes
- ✗Treating maximum revenue as the whole objective and ignoring retention
- ✗Believing logged prices alone reveal what an unshown price would have earned
- ✗Randomizing per impression instead of per user, so one buyer sees jumping prices
Follow-up questions
- →How large a traffic slice would you give to randomized exploration, and why?
- →Which guardrail metric would stop a pricing A/B-test experiment early?
SeniorDesignOccasionalA delivery service completes about 500k orders a day and shows an ETA to the customer before they order. A late delivery costs several times more in refunds and churn than an equally early one. The prediction must return inside 50 ms at p99, and you have route distance, live traffic, courier supply, restaurant preparation history and time-of-day signals. Walk through the design — which features you build and how you serve them inside the latency budget, why the training loss cannot be symmetric, how you evaluate when RMSE says one thing and the business says another, and what happens to your training data once couriers and customers react to the ETA you display.
SeniorDesignOccasional
A delivery service completes about 500k orders a day and shows an ETA to the customer before they order. A late delivery costs several times more in refunds and churn than an equally early one. The prediction must return inside 50 ms at p99, and you have route distance, live traffic, courier supply, restaurant preparation history and time-of-day signals. Walk through the design — which features you build and how you serve them inside the latency budget, why the training loss cannot be symmetric, how you evaluate when RMSE says one thing and the business says another, and what happens to your training data once couriers and customers react to the ETA you display.
Serve precomputed route, traffic, supply and prep features to hold p99. Train a quantile loss at an upper quantile so lateness costs more than earliness, and score on that asymmetric cost, not RMSE. The shown ETA contaminates future data.
Common mistakes
- ✗Using a symmetric loss when lateness and earliness cost the business differently
- ✗Judging the model by RMSE instead of the asymmetric business cost
- ✗Treating logged delivery times as unaffected by the ETA that was displayed
Follow-up questions
- →How would you pick which upper quantile the pinball loss — an asymmetric quantile objective — targets?
- →How would you keep an unbiased data slice despite the feedback loop the shown ETA creates?
SeniorDesignOccasionalYou are designing the recommendation feed on a marketplace home page. The catalogue holds about 50M items and roughly 20M users open the feed daily; the p99 server-side budget for assembling one feed is about 150 ms. Around 30% of impressions go to users with fewer than five past interactions, and about 10% of the catalogue is new every week. Click, add-to-cart and purchase logs are available, plus item titles, descriptions and images. Walk through the metric you optimize online and offline, the split between candidate generation and ranking, how you handle cold start for both new users and new items, and how the design stays inside the serving cost envelope.
SeniorDesignOccasional
You are designing the recommendation feed on a marketplace home page. The catalogue holds about 50M items and roughly 20M users open the feed daily; the p99 server-side budget for assembling one feed is about 150 ms. Around 30% of impressions go to users with fewer than five past interactions, and about 10% of the catalogue is new every week. Click, add-to-cart and purchase logs are available, plus item titles, descriptions and images. Walk through the metric you optimize online and offline, the split between candidate generation and ranking, how you handle cold start for both new users and new items, and how the design stays inside the serving cost envelope.
Optimize an online metric such as purchases per session, using an offline ranking metric as its proxy. Serve in two stages — cheap candidate generation, then a heavier ranker — to hold 150 ms. Cold start leans on content, not history.
Common mistakes
- ✗Ranking the whole catalogue per request instead of generating candidates first
- ✗Treating the offline ranking metric as the goal rather than a proxy for the online one
- ✗Assuming cold-start users and items can be served from interaction history alone
Follow-up questions
- →How would you break the feedback loop where the feed only shows what it already ranks highly?
- →Which offline ranking metric — say NDCG, a graded ranking metric — would you trust for feed quality?
SeniorDesignOccasionalA classifieds platform holds about 100M active listings with text and photos, and roughly 1M new listings arrive a day that must be checked for near-duplicates within minutes of posting. Comparing every pair is out of the question — 100M items is on the order of 10^16 pairs. There is no labelled duplicate set and nobody will hand-label one. Walk through the design — how you represent text and images, how you cut the candidate space so you never score all pairs, what you retrieve with and how you choose the match threshold, and how you measure whether the system works with no ground truth available.
SeniorDesignOccasional
A classifieds platform holds about 100M active listings with text and photos, and roughly 1M new listings arrive a day that must be checked for near-duplicates within minutes of posting. Comparing every pair is out of the question — 100M items is on the order of 10^16 pairs. There is no labelled duplicate set and nobody will hand-label one. Walk through the design — how you represent text and images, how you cut the candidate space so you never score all pairs, what you retrieve with and how you choose the match threshold, and how you measure whether the system works with no ground truth available.
Embed each modality, then cut the pair space by blocking and an ANN index so only candidates are rescored. Pick the threshold on sampled precision. Without labels, mine weak positives from known reposts and track a business proxy.
Common mistakes
- ✗Treating all-pairs comparison as a compute problem instead of avoiding it entirely
- ✗Assuming a labelled duplicate set exists rather than building evaluation from sampling
- ✗Fixing the threshold from the score distribution instead of measured precision at that cut
Follow-up questions
- →How does recall of the ANN index — approximate nearest neighbour search — bound overall recall?
- →How would a two-tower retrieval encoder change candidate generation versus hand-built blocking keys?
SeniorDesignOccasionalYou must design the ML platform for a 30-person data-science team running about 40 models in production. Today several teams recompute the same customer features with slightly different logic, there is no shared experiment history, and every deploy is done by hand by whoever trained the model. Your platform team is three engineers, so you cannot build everything. Walk through the platform you would put in place — how training and serving come to share one feature definition, how experiments become reproducible, what a model registry with CI for models must gate before promotion, how serving is handled, and where you would buy rather than build.
SeniorDesignOccasional
You must design the ML platform for a 30-person data-science team running about 40 models in production. Today several teams recompute the same customer features with slightly different logic, there is no shared experiment history, and every deploy is done by hand by whoever trained the model. Your platform team is three engineers, so you cannot build everything. Walk through the platform you would put in place — how training and serving come to share one feature definition, how experiments become reproducible, what a model registry with CI for models must gate before promotion, how serving is handled, and where you would buy rather than build.
Make a feature store the single definition training and serving share, killing training-serving skew. Add experiment tracking for reproducible runs, a model registry with CI over data and models, and automated serving. Buy commodity, build domain.
Common mistakes
- ✗Treating training-serving skew as a serving bug instead of a definition problem
- ✗Gating promotion with unit tests on code while never testing data or the model
- ✗Building every platform component in-house with a three-engineer team
Follow-up questions
- →Which model CI checks would you run on data rather than on code, and why?
- →How do you migrate 40 live models onto a shared feature store without a freeze?
SeniorDesignOccasionalYou own matching between resumes and vacancies at a job board. There are about 5M resumes and about 300k open vacancies refreshed daily, and a recruiter only ever looks at the top 50 candidates per vacancy. Your only labels are recruiter invitations and hire outcomes — sparse, and biased by who recruiters happened to look at before. Brand-new vacancies arrive with zero interaction history, and the product is legally exposed on fairness. Walk through the retrieval and ranking architecture you would build, where the labels come from and how they mislead, how you cover cold start for new vacancies and new resumes, and how you would constrain and verify fairness.
SeniorDesignOccasional
You own matching between resumes and vacancies at a job board. There are about 5M resumes and about 300k open vacancies refreshed daily, and a recruiter only ever looks at the top 50 candidates per vacancy. Your only labels are recruiter invitations and hire outcomes — sparse, and biased by who recruiters happened to look at before. Brand-new vacancies arrive with zero interaction history, and the product is legally exposed on fairness. Walk through the retrieval and ranking architecture you would build, where the labels come from and how they mislead, how you cover cold start for new vacancies and new resumes, and how you would constrain and verify fairness.
Retrieve with a two-tower encoder, then rerank the shortlist with a cross-encoder. Invitation and hire labels are sparse and biased by past recruiter choices. Serve cold start from content features and monitor outcome parity, since proxies leak.
Common mistakes
- ✗Running one heavy ranking model over the whole corpus instead of retrieving first
- ✗Reading recruiter invitations as objective relevance rather than logged behaviour
- ✗Assuming that removing protected attributes removes the disparity itself
Follow-up questions
- →How do you build negatives when the only signal is a recruiter invitation?
- →Which fairness metric would you monitor for the recruiter top 50 and why?
SeniorDesignOccasionalYou are designing search ranking for a classifieds site. About 100M listings are active, search runs at roughly 500 queries per second, and the p95 budget is about 200 ms. There is no editorial relevance dataset and no budget to label at scale — you only have logs of clicks and contact-seller events, and results are shown to the user as a single ranked list. Walk through where your relevance labels come from, how you train the ranker with the learning-to-rank family of ranking objectives and which of pointwise, pairwise or listwise you pick, the offline ranking metric you gate on, and the online metric you ship on.
SeniorDesignOccasional
You are designing search ranking for a classifieds site. About 100M listings are active, search runs at roughly 500 queries per second, and the p95 budget is about 200 ms. There is no editorial relevance dataset and no budget to label at scale — you only have logs of clicks and contact-seller events, and results are shown to the user as a single ranked list. Walk through where your relevance labels come from, how you train the ranker with the learning-to-rank family of ranking objectives and which of pointwise, pairwise or listwise you pick, the offline ranking metric you gate on, and the online metric you ship on.
Labels come from implicit feedback — clicks and contacts — corrected for position bias, plus a small assessor set. Train a pairwise or listwise ranker — the product is an ordered list. Gate offline on NDCG, ship on contact rate in an A/B test.
Common mistakes
- ✗Taking raw clicks as unbiased relevance and skipping position-bias correction
- ✗Optimizing a pointwise objective when the delivered product is a ranked list
- ✗Shipping on an offline ranking metric without confirming the online metric
Follow-up questions
- →How would you estimate the position-bias curve without degrading the results users see?
- →When does a listwise objective beat a pairwise one in the learning-to-rank family?
SeniorDesignOccasionalA company wants an LLM support assistant over an internal knowledge base of about 50k documents, a few percent of which change every week. It will serve roughly 20k conversations a day under a hard ceiling of a few cents per conversation. Every answer must cite the document it came from, and the user must always be able to reach a human agent. Walk through the design — whether you fine-tune on the knowledge base or retrieve from it and why, what stops the assistant from inventing an answer, when a conversation is escalated to a human, how you evaluate the system, and where the cost per conversation actually goes.
SeniorDesignOccasional
A company wants an LLM support assistant over an internal knowledge base of about 50k documents, a few percent of which change every week. It will serve roughly 20k conversations a day under a hard ceiling of a few cents per conversation. Every answer must cite the document it came from, and the user must always be able to reach a human agent. Walk through the design — whether you fine-tune on the knowledge base or retrieve from it and why, what stops the assistant from inventing an answer, when a conversation is escalated to a human, how you evaluate the system, and where the cost per conversation actually goes.
Retrieve rather than fine-tune — weekly-changing facts need a reindex, not a retrain. Answers are grounded in retrieved passages and cite them, and weak retrieval triggers a refusal or handoff. Evaluate retrieval and groundedness separately.
Common mistakes
- ✗Fine-tuning to inject facts that change weekly instead of retrieving them
- ✗Treating a confidently worded answer as grounded without checking retrieved passages
- ✗Scoring only final answers, which leaves retrieval failures invisible
Follow-up questions
- →How do you size the passage count the RAG — retrieval-augmented generation — pipeline puts in context?
- →What would you still fine-tune for once retrieval already handles the changing facts?
