Product Analytics
Metric design and selection, unit economics, prioritization, and structured investigation.
16 questions
JuniorDesignVery commonEstimate the number of houses in Moscow — give an upper and a lower bound. Devise an estimation method that produces an upper threshold and a lower threshold for the count of residential buildings in Moscow.
Estimate the number of houses in Moscow — give an upper and a lower bound. Devise an estimation method that produces an upper threshold and a lower threshold for the count of residential buildings in Moscow.
Anchor on population (and/or area) and reason by bounds. Lower bound: assume everyone lives in the largest plausible high-rises — population divided by the maximum people per building gives the fewest buildings. Upper bound: the smallest buildings — population over the minimum people per building, or area over a minimum footprint. State every assumption.
Common mistakes
- ✗Giving one number with no method or assumptions
- ✗Failing to separate the upper from the lower bound
- ✗Not anchoring on a population or area figure
Follow-up questions
- →Which anchor — population or area — gives a tighter bound?
- →How would you sanity-check the two bounds against each other?
JuniorDesignVery commonA key metric dropped 30% overnight. How do you structure the root-cause search? A conversion or other key metric suddenly fell about 30%. Walk through, as an ordered set of blocks, what could have happened and how you would localize the cause.
A key metric dropped 30% overnight. How do you structure the root-cause search? A conversion or other key metric suddenly fell about 30%. Walk through, as an ordered set of blocks, what could have happened and how you would localize the cause.
Work in blocks. First the data itself: seasonality/holidays, broken counters or logging, storage/load gaps; verify the numerator and denominator absolutes. Then localize the segment: traffic source, device/browser/region, a specific funnel step.
Common mistakes
- ✗Jumping to a fix before checking data quality
- ✗Skipping segment localization to find where it dropped
- ✗Ignoring external factors and parallel experiments
Follow-up questions
- →How would you tell a broken counter from a real drop?
- →Which segment cut would you check first and why?
MiddleDesignVery commonYou join a product with no clear metrics. How do you choose them from scratch? As a new product analyst you find the metrics in disarray. Describe your approach to picking a coherent metric set.
You join a product with no clear metrics. How do you choose them from scratch? As a new product analyst you find the metrics in disarray. Describe your approach to picking a coherent metric set.
Start from goals, not from data: business model → core metrics → the metrics you can actually influence → targets for them. Structure it with a framework (AARRR, HEART, or a metric hierarchy/pyramid) and separate product metrics from growth metrics. Be able to explain each metric — e.g.
Common mistakes
- ✗Starting from available data instead of business goals
- ✗Tracking many redundant metrics with no hierarchy
- ✗Mixing product metrics with growth metrics undistinguished
Follow-up questions
- →How does AARRR differ from a metric pyramid here?
- →Why separate metrics you can influence from the rest?
JuniorTheoryCommonLoyalty members buy 30% more; half are enrolled. What if you auto-enroll everyone?
Loyalty members buy 30% more; half are enrolled. What if you auto-enroll everyone?
You cannot promise the 30% lift — the link is correlation, not proven causation. Frequent, motivated buyers may simply self-select into the loyalty program, so the program reflects their behavior rather than causing it. Auto-enrolling everyone need not raise spend at all.
Common mistakes
- ✗Reading the correlation as proven causation
- ✗Ignoring self-selection of frequent buyers into the program
- ✗Skipping an A/B test to isolate the program's real effect
Follow-up questions
- →What common cause could drive both enrollment and spending?
- →How would you design the A/B test to measure the lift?
JuniorDesignCommonA new feature ships with no metrics attached. Name the ONE metric you would track first, and defend why it — not a vanity count — is the right first choice for judging the feature.
A new feature ships with no metrics attached. Name the ONE metric you would track first, and defend why it — not a vanity count — is the right first choice for judging the feature.
Pick the metric that captures the feature's core success — usually adoption or its intended outcome, as a rate over the eligible audience, not a raw count. Defend it by tying it to why the feature was built: it must move when the feature works and stay flat when users get no value from it.
Common mistakes
- ✗Tracking a raw count that only grows over time
- ✗Picking a metric unrelated to the feature's goal
- ✗Choosing several metrics instead of one primary one
Follow-up questions
- →Why prefer a rate over a raw usage count here?
- →What guardrail would you watch alongside this one metric?
MiddleDesignCommonHow do you prioritize a product backlog as an analyst? Given a backlog, identify the company's challenges, propose ideas, and decide what to build first and what to drop.
How do you prioritize a product backlog as an analyst? Given a backlog, identify the company's challenges, propose ideas, and decide what to build first and what to drop.
First name the company's real challenges and candidate solutions, then pick a North Star metric to judge against. Estimate each feature's impact in money or in that target metric, weigh it against development cost (an effort-versus-impact framing like RICE/ICE), and rank accordingly.
Common mistakes
- ✗Prioritizing by request order or stakeholder volume
- ✗Ranking by build cost alone, ignoring impact
- ✗Refusing to drop low-impact items from the backlog
Follow-up questions
- →How would RICE change your ranking versus raw impact?
- →How do you estimate a feature's impact before building it?
MiddleDesignCommonThe CEO asks 'are we doing well?' and wants it on one slide. Which three numbers do you put up, and why exactly those three? Assume a subscription product and a single slide only.
The CEO asks 'are we doing well?' and wants it on one slide. Which three numbers do you put up, and why exactly those three? Assume a subscription product and a single slide only.
Put one number per layer of the story: growth (active or new users), retention (retained users or revenue retention), and money (revenue or margin). Together they answer are we growing, are we keeping people, and does it pay — and each guards the others, exposing growth that is quietly funded by churn.
Common mistakes
- ✗Filling the slide with many metrics instead of three
- ✗Showing only growth, hiding retention and revenue
- ✗Choosing three numbers that all say the same thing
Follow-up questions
- →Why include retention right next to a growth number?
- →How could three correlated metrics mislead the CEO?
MiddleDesignCommonTwo dashboards report a different DAU for the same day. How do you find out which one is right? Both claim to show daily active users for yesterday and disagree by about 8%. Walk through how you would reconcile them.
Two dashboards report a different DAU for the same day. How do you find out which one is right? Both claim to show daily active users for yesterday and disagree by about 8%. Walk through how you would reconcile them.
Reconcile definitions, not values. Compare how each dashboard defines active (event versus login), the timezone/day boundary, user deduplication, and bot filtering, then the data source and sampling. Reproduce the count from raw events; the right number is the one whose definition matches the metric you intend.
Common mistakes
- ✗Assuming the higher (or lower) number is automatically right
- ✗Comparing values before comparing metric definitions
- ✗Ignoring timezone and dedup as sources of the gap
Follow-up questions
- →How could a timezone boundary alone create an 8% gap?
- →Which single source of truth would you reconcile against?
MiddleTheoryCommonA manager proposes 'average time on site' — how do you judge if it's a good metric?
A manager proposes 'average time on site' — how do you judge if it's a good metric?
Judge it against quality criteria: it must tie to a business goal, be actionable (you know whether a ±5% move is good or bad and what to do), be measurable (handle anonymous users, map user↔account), be sensitive (reacts to real changes above noise), and be decomposable.
Common mistakes
- ✗Accepting a metric without checking it ties to a goal
- ✗Assuming more time on site is always good
- ✗Ignoring actionability and sensitivity criteria
Follow-up questions
- →For which product would 'time on site' be a bad metric?
- →What makes a metric actionable in practice?
MiddleDesignCommonA company launched paid acquisition. Should it keep advertising? Decide whether to keep paid acquisition running, and explain what you would do if it had not been launched yet.
A company launched paid acquisition. Should it keep advertising? Decide whether to keep paid acquisition running, and explain what you would do if it had not been launched yet.
Compare CAC against the LTV of acquired users (or ROI versus benchmarks). Compute LTV from profit, not revenue, and discount future cash flows. CAC > LTV is not always fatal — early-stage businesses may pay to grow. Also weigh cannibalization of organic signups.
Common mistakes
- ✗Computing LTV from revenue instead of profit
- ✗Treating CAC > LTV as always a stop signal
- ✗Ignoring cannibalization of organic users
Follow-up questions
- →How would a geo test isolate paid acquisition's incremental effect?
- →Why discount future cash flows when computing LTV?
JuniorDesignOccasionalFor a marketplace where good search means 'the user found it fast', how would you measure whether search is good? Design the success metrics, taking 'found it fast' as the working definition of good.
For a marketplace where good search means 'the user found it fast', how would you measure whether search is good? Design the success metrics, taking 'found it fast' as the working definition of good.
Turn 'found it fast' into measurable proxies: search success rate (a session ending in a click or purchase), time or queries to the first meaningful click, and zero-result or reformulation rate. Pick a primary — success rate — and watch the rest as guardrails so speed is not bought by worse relevance.
Common mistakes
- ✗Measuring raw search volume as if more is better
- ✗Using one metric with no guardrail on relevance
- ✗Equating 'fast' with latency, ignoring finding the item
Follow-up questions
- →Why can a high query count signal failure, not engagement?
- →How would you tell fast success from fast giving-up?
JuniorTheoryOccasionalWhy is 'total registered users' a vanity metric, and what would you track instead?
Why is 'total registered users' a vanity metric, and what would you track instead?
It only ever grows and cannot fall, so it never tells you whether the product is doing better or worse today — it is not actionable. Track metrics that can move both ways and reflect current value: active users (DAU/WAU), retention, and conversion, which react to real product changes.
Common mistakes
- ✗Judging a metric by how impressive its number looks
- ✗Assuming a monotonically growing number signals health
- ✗Reporting cumulative totals instead of current activity
Follow-up questions
- →Name another cumulative total that behaves as a vanity metric.
- →How does retention reveal what registered-users hides?
JuniorDesignOccasionalA team tracks a metric with two separate bar series. How would you improve this chart? The company watches a key indicator by plotting two bar series side by side. What is wrong with this way of showing it, and what would you change?
A team tracks a metric with two separate bar series. How would you improve this chart? The company watches a key indicator by plotting two bar series side by side. What is wrong with this way of showing it, and what would you change?
The actual metric is not on the chart — two raw bar series force the reader to compute the conversion mentally. Replace them with the single computed metric (the conversion itself), shown as a line or one bar over time so the trend reads instantly.
Common mistakes
- ✗Leaving the reader to compute the metric from raw bars
- ✗Adding chart decoration instead of showing the metric
- ✗Dropping the time axis a key metric needs for trend
Follow-up questions
- →When is a line chart better than bars for this metric?
- →How would you annotate a known cause of a dip?
MiddleDesignOccasionalTwo elevators run different unknown algorithms — how do you measure which is better? A building has two elevators driven by two different, opaque dispatch algorithms. You want to pick one to standardize on. How would you measure and compare how well each performs?
Two elevators run different unknown algorithms — how do you measure which is better? A building has two elevators driven by two different, opaque dispatch algorithms. You want to pick one to standardize on. How would you measure and compare how well each performs?
Define success metrics first: average and 95th-percentile wait time, ride/travel time, peak-hour throughput, energy use, and fairness across floors. Pick a primary metric. Instrument both elevators over comparable loads, ideally an A/B-style comparison, and compare the metric distributions, not just averages.
Common mistakes
- ✗Comparing on subjective feel instead of measured metrics
- ✗Not controlling for time-of-day and load between elevators
- ✗Comparing averages only, ignoring the distribution and percentiles
Follow-up questions
- →Why report the 95th-percentile wait, not just the average?
- →How would you control for floor-level traffic differences?
SeniorDesignOccasionalA stakeholder asks 'should we build a mobile app?' Frame it as an analyst: what evidence would change your answer, and what would you refuse to promise? State the decision, the evidence you would gather, and the outcomes you would not commit to on the data alone.
A stakeholder asks 'should we build a mobile app?' Frame it as an analyst: what evidence would change your answer, and what would you refuse to promise? State the decision, the evidence you would gather, and the outcomes you would not commit to on the data alone.
Frame it as a decision, not a yes or no. Evidence that moves me: mobile-web share and its retention gap, demand signals, and a mobile cohort's unit economics versus build and upkeep cost. I won't promise a specific revenue or retention lift — a causal claim the data cannot back without an experiment or staged rollout.
Common mistakes
- ✗Answering yes or no without stating the evidence
- ✗Promising a revenue lift from correlational usage data
- ✗Ignoring build and maintenance cost against the payoff
Follow-up questions
- →How would a staged rollout turn a correlation into a causal estimate?
- →Which cost would you weigh against the projected mobile LTV?
JuniorDesignRareEstimate how many pizzas are delivered in Moscow on a Friday night — and say which assumption dominates your error. Give a method, not a single number, and name the input whose uncertainty most drives the estimate.
Estimate how many pizzas are delivered in Moscow on a Friday night — and say which assumption dominates your error. Give a method, not a single number, and name the input whose uncertainty most drives the estimate.
Anchor on population: ~12M people → households → the share ordering delivery pizza on a Friday night → pizzas per order. Multiply through, stating each assumption. Then name the dominant error term — usually the order-rate share, a guess spanning an order of magnitude, not the well-known population.
Common mistakes
- ✗Giving a single number with no decomposition or assumptions
- ✗Treating every assumption as equally uncertain
- ✗Anchoring on a figure you cannot actually justify
Follow-up questions
- →How would you narrow the order-rate share with a quick check?
- →Which is better known here — population or pizzas per order?
