Recommender Systems
Collaborative/content/hybrid filtering, matrix factorization, cold start, candidate generation + ranking, two-tower models, NDCG, position bias and feedback loops.
11 questions
JuniorTheoryVery commonCollaborative, content-based and hybrid filtering — what data does each need?
Collaborative, content-based and hybrid filtering — what data does each need?
Collaborative filtering needs only the interaction matrix — it suggests what similar users liked, so new users and items break it. Content-based needs item features and a profile, covering new items but never leaving known taste. Hybrids blend both.
Common mistakes
- ✗Assuming collaborative filtering can score a brand-new item with no interactions
- ✗Thinking content-based filtering discovers taste outside a user's history
- ✗Treating a hybrid as a fallback switch rather than a blend of two signals
Follow-up questions
- →Which approach would you ship first for a catalogue that turns over weekly, and why?
- →How does a hybrid reweight the two signals as a user accumulates interactions?
JuniorTheoryVery commonWhat are the three flavours of cold start, and one practical fix for each?
What are the three flavours of cold start, and one practical fix for each?
New user — no history, so serve popularity or onboarding picks. New item — no interactions, so score it from content features. New system — no matrix, so bootstrap from content rules. All three break collaborative filtering, not content signals.
Common mistakes
- ✗Believing matrix factorization yields a usable vector for a zero-interaction item
- ✗Attacking cold start with hyperparameter tuning instead of a different signal source
- ✗Ignoring system cold start when launching into an empty interaction log
Follow-up questions
- →How would you evaluate a cold-start fix when the affected users are a tiny slice?
- →What exploration budget would you give brand-new items, and for how long?
JuniorTheoryCommonRanking metrics NDCG@k, MAP, recall@k and hit-rate — what does each reward?
Ranking metrics NDCG@k, MAP, recall@k and hit-rate — what does each reward?
Recall@k rewards relevant items in the top k, order ignored. Hit-rate asks only whether one landed there. MAP averages precision at each hit, so order counts. NDCG adds graded relevance and a positional discount, making it the ranking default.
Common mistakes
- ✗Treating recall@k as order-sensitive
- ✗Assuming relevance labels must be binary, so graded gain is unavailable
- ✗Reporting hit-rate as if it measured how many relevant items were retrieved
Follow-up questions
- →When would you prefer recall@k over NDCG for evaluating a candidate generator?
- →How does the choice of k interact with the number of slots the interface shows?
MiddleTheoryCommonWhat do latent factors mean, and how does alternating least squares treat clicks?
What do latent factors mean, and how does alternating least squares treat clicks?
Latent factors are learned coordinates with no fixed meaning — user and item vectors whose dot product predicts affinity. Implicit feedback has no true negatives, so an unobserved cell is a low-confidence negative and a click a weighted positive.
Common mistakes
- ✗Reading a single latent dimension as a named genre or attribute
- ✗Treating an unobserved cell as a confident dislike rather than weak evidence
- ✗Dropping unobserved cells entirely, so the model only ever sees positives
Follow-up questions
- →How should the confidence weight grow as a user's click count on one item rises?
- →Why does alternating least squares parallelise better than gradient descent here?
MiddleTheoryCommonIn a two-tower retrieval model, why must the towers stay separate, and what is lost?
In a two-tower retrieval model, why must the towers stay separate, and what is lost?
They stay separate so item embeddings are built offline and indexed for approximate neighbour search, leaving only the user tower per request. The price is no user-item cross feature, so interactions a cross-feature ranker sees stay invisible.
Common mistakes
- ✗Believing the final dot product recovers user-item cross features
- ✗Scoring the whole catalogue at request time instead of querying a neighbour index
- ✗Expecting two-tower retrieval to replace the downstream ranking stage
Follow-up questions
- →How stale can the offline item index get before retrieval quality visibly drops?
- →How would you mine hard negatives for the retrieval loss beyond in-batch negatives?
SeniorDesignCommonYou own the home feed of a marketplace with fifty million items and five million daily users. The feed must return thirty ranked items inside a 150 ms end-to-end budget, and the ranking model your team wants to ship is a heavy cross-feature network costing roughly one millisecond per item scored. Design the system as two stages — an approximate nearest-neighbour candidate generator followed by that ranker. Explain why a single stage cannot meet the budget, how many items each stage hands on, what share of the latency budget you give each stage, which model family fits each stage, and how you keep the two stages from optimising against each other.
You own the home feed of a marketplace with fifty million items and five million daily users. The feed must return thirty ranked items inside a 150 ms end-to-end budget, and the ranking model your team wants to ship is a heavy cross-feature network costing roughly one millisecond per item scored. Design the system as two stages — an approximate nearest-neighbour candidate generator followed by that ranker. Explain why a single stage cannot meet the budget, how many items each stage hands on, what share of the latency budget you give each stage, which model family fits each stage, and how you keep the two stages from optimising against each other.
One stage cannot work — the heavy ranker cannot touch fifty million items, and a cheap model cannot rank well. Retrieval cuts them to a few hundred in tens of milliseconds via a neighbour index over two-tower embeddings; the ranker gets the rest.
Common mistakes
- ✗Sizing the candidate set without checking it against the ranker's per-item cost
- ✗Assuming a cheaper single model can both scan the catalogue and rank it well
- ✗Training retrieval on a distribution the ranker never actually receives
Follow-up questions
- →How would you detect that retrieval has become the quality bottleneck, not the ranker?
- →What changes in this design when the latency budget is cut from 150 ms to 50 ms?
MiddleDesignOccasionalYour marketplace feed ranks purely by predicted click-through rate. Merchandising reports that the top twenty slots are dominated by three large sellers and one product category, small sellers are churning off the platform, and users describe the feed as repetitive — yet click-through rate itself is at an all-time high. You have to design the ranking objective that replaces pure predicted click-through rate. Define what you actually optimise, how diversity and seller exposure enter the objective, what you refuse to trade away, and how you would verify before a full rollout that the trade is worth making.
Your marketplace feed ranks purely by predicted click-through rate. Merchandising reports that the top twenty slots are dominated by three large sellers and one product category, small sellers are churning off the platform, and users describe the feed as repetitive — yet click-through rate itself is at an all-time high. You have to design the ranking objective that replaces pure predicted click-through rate. Define what you actually optimise, how diversity and seller exposure enter the objective, what you refuse to trade away, and how you would verify before a full rollout that the trade is worth making.
Optimise long-run value, not per-impression clicks — relevance plus a diversity term and a seller-exposure constraint in the re-ranker. Never trade the relevance floor. Verify by A/B test on user and seller retention, since clicks are what it spends.
Common mistakes
- ✗Optimising a per-impression proxy while the business goal is long-run retention
- ✗Bolting diversity on as a post-filter instead of putting it in the objective
- ✗Reading the experiment on the very metric the change deliberately spends
Follow-up questions
- →How would you set the weight on the diversity term without hand-tuning it forever?
- →What guardrail metric would stop the rollout even if retention looks positive?
MiddleDebuggingOccasionalOffline NDCG rose but online click-through rate fell — fix this evaluation script.
Offline NDCG rose but online click-through rate fell — fix this evaluation script.
It scores the full log including trained-on rows, splits randomly so future sessions leak back, and trusts logged clicks though only what the old ranker exposed was seen. Split by time, score held-out sessions, and reweight by inverse propensity.
Open full question →Common mistakes
- ✗Random-splitting interaction logs instead of splitting by time
- ✗Scoring the metric on rows the model was trained on
- ✗Treating logged clicks as unbiased relevance labels
Follow-up questions
- →How would you build a counterfactual estimator that better predicts online lift?
- →Which of the three flaws would you expect to inflate offline NDCG the most?
MiddleTheoryOccasionalWhy is a click at rank one weak evidence of relevance, and how do you debias logs?
Why is a click at rank one weak evidence of relevance, and how do you debias logs?
Rank one draws more attention, so clicks there reflect exposure as much as relevance, and raw-click training repeats yesterday's order. Debias by estimating examination propensity per position and weighting clicks by its inverse, from randomized slots.
Common mistakes
- ✗Assuming position bias averages out given enough logged sessions
- ✗Weighting clicks by position instead of by inverse examination propensity
- ✗Estimating propensities from the same logs the production ranker produced
Follow-up questions
- →How large must the randomized slot budget be before propensity estimates are stable?
- →How does inverse propensity weighting change the variance of an offline estimate?
SeniorDesignOccasionalSix months after launch your recommender serves from roughly two hundred items out of a four-hundred-thousand item catalogue. It has been retrained weekly on its own click logs the entire time. Impressions land almost entirely on those two hundred items, new inventory almost never surfaces, sellers of unshown items are leaving, and yet every offline metric computed on those logs looks excellent and perfectly stable. Revenue is flat rather than falling, so leadership will not approve a change that costs measurable short-term revenue. Design how you break the loop — where the exploration comes from, how you bound its cost, and how you prove the loop is genuinely broken rather than merely disturbed for a week.
Six months after launch your recommender serves from roughly two hundred items out of a four-hundred-thousand item catalogue. It has been retrained weekly on its own click logs the entire time. Impressions land almost entirely on those two hundred items, new inventory almost never surfaces, sellers of unshown items are leaving, and yet every offline metric computed on those logs looks excellent and perfectly stable. Revenue is flat rather than falling, so leadership will not approve a change that costs measurable short-term revenue. Design how you break the loop — where the exploration comes from, how you bound its cost, and how you prove the loop is genuinely broken rather than merely disturbed for a week.
The logs hold only what the model showed, so retraining is self-confirming. Break it with a randomized exposure slice whose propensities are logged and reweight training, making unshown stock scoreable. Cap the slice and read coverage on a holdback.
Common mistakes
- ✗Trusting offline metrics computed on logs the model itself generated
- ✗Adding exploration without logging the propensities it produces
- ✗Judging success on revenue alone while catalogue coverage stays collapsed
Follow-up questions
- →What size of exploration slice keeps the revenue cost inside a single-digit percentage?
- →How long must coverage hold after you stop exploring before you call the loop broken?
JuniorTheoryRareUser-based versus item-based kNN collaborative filtering — which one scales?
User-based versus item-based kNN collaborative filtering — which one scales?
Item-based kNN scales. Items are fewer and more stable than users, so the item-item neighbour table is precomputed offline and reused for days, while user vectors shift every session. Cosine similarity over mean-centred vectors is the usual choice.
Common mistakes
- ✗Assuming user-user and item-item cost the same because the matrix is the same
- ✗Using raw counts without mean-centring, so heavy users dominate every similarity
- ✗Recomputing neighbours per request instead of serving a precomputed table
Follow-up questions
- →How does item popularity skew item-item similarity, and how would you damp it?
- →At what catalogue size does the user-based variant become the cheaper option?
