Metric Root Cause
The "why did X drop 20%" case family — artifact versus real, slicing order, mix shift, and external shocks.
15 questions
JuniorDesignVery commonA dashboard shows a key metric down 20% overnight and the team wants a fix. Before forming a single hypothesis, what do you verify first, and why in that order? You have minutes, and cannot inspect everything at once.
A dashboard shows a key metric down 20% overnight and the team wants a fix. Before forming a single hypothesis, what do you verify first, and why in that order? You have minutes, and cannot inspect everything at once.
First rule out a data artifact: confirm the definition is unchanged, counters and logging still fire, and yesterday's data is fully loaded. Pipeline and logging breaks are the commonest, cheapest cause, so clearing them before a product hypothesis avoids chasing a ghost.
Common mistakes
- ✗Jumping to a hypothesis or a fix before validating the data
- ✗Checking segments and releases before confirming the number is real
- ✗Assuming an overnight move is too big to be an artifact
Follow-up questions
- →How would you tell a broken counter from a real drop?
- →Which data check gives the most certainty for the least time?
MiddleDesignVery commonDAU dropped 20% overnight with no known release. Before you touch product hypotheses, how do you separate a genuine drop from a logging or tracking artifact? State the concrete checks and what result would convince you it is real.
DAU dropped 20% overnight with no known release. Before you touch product hypotheses, how do you separate a genuine drop from a logging or tracking artifact? State the concrete checks and what result would convince you it is real.
Cross-check the metric against an independent source that shares no logging path — server logs, payments, or a backend event count. If that source also dropped, it is real; if only the tracked metric fell, suspect a broken SDK, tag, or pipeline. Also confirm the day loaded fully.
Common mistakes
- ✗Trusting one tracked source instead of an independent cross-check
- ✗Assuming tracking is fine because no release shipped
- ✗Confusing a reproducible query with a verified real drop
Follow-up questions
- →Which independent source would you trust most, and why?
- →How would a partial SDK failure look versus a full one?
MiddleDesignVery commonA conversion rate is down 20% overall, yet when you slice by any single dimension every segment's rate is flat or even up. How is that possible, and how do you find the mix shift behind it?
A conversion rate is down 20% overall, yet when you slice by any single dimension every segment's rate is flat or even up. How is that possible, and how do you find the mix shift behind it?
The overall rate is a weighted average, so it can fall while every segment holds if the mix shifted toward lower-converting ones. Find it by comparing each segment's share before and after, then recomputing the total on the old mix; if that is flat, the mix moved.
Common mistakes
- ✗Believing the total cannot move while every segment holds
- ✗Ignoring how each segment's share of the mix changed
- ✗Reporting the simple average of segments as the true total
Follow-up questions
- →How does the aggregation trap known as Simpson's paradox produce this?
- →How would you quantify how much of the drop the mix explains?
JuniorDesignCommonA stakeholder messages 'conversion is down' and asks you to dig in. Before you open any query, what do you ask them — and what turns a scary-sounding number into a real anomaly worth investigating?
A stakeholder messages 'conversion is down' and asks you to dig in. Before you open any query, what do you ask them — and what turns a scary-sounding number into a real anomaly worth investigating?
Scope it before touching data: which metric exactly, over what window, against what baseline, and how large in absolute terms. A move is an anomaly only if it exceeds normal daily and weekly variation — a 2% wobble inside noise or a routine weekend dip is expected, not an incident.
Common mistakes
- ✗Investigating before confirming the change exceeds normal variation
- ✗Not pinning down the exact metric, window, and baseline
- ✗Treating any reported dip as a confirmed incident
Follow-up questions
- →How would you set the threshold for what counts as an anomaly?
- →How do weekly and seasonal patterns change that threshold?
JuniorDesignCommonYou have confirmed a metric drop is real and now must localize it. You cannot slice by every dimension at once — how do you choose which cuts to run, and in what order, to find where the drop lives?
You have confirmed a metric drop is real and now must localize it. You cannot slice by every dimension at once — how do you choose which cuts to run, and in what order, to find where the drop lives?
Slice one dimension at a time, ordered by likelihood: platform and app version, then traffic source, geography, and funnel step. After each cut, ask whether the drop sits in one slice or spreads evenly — concentration points to a specific cause, even spread to something global.
Common mistakes
- ✗Slicing by every dimension at once instead of one at a time
- ✗Not asking whether the drop is concentrated or spread
- ✗Starting from tiny segments before the high-level cuts
Follow-up questions
- →How does concentrated versus uniform spread change your next step?
- →What would you check if no single slice explains the drop?
MiddleDesignCommonOne incident is a sharp cliff at 14:00 on a single day; another is a gradual 3-week slide. Assuming both are real, what does each shape tell you about the likely cause, and how does it change where you look first?
One incident is a sharp cliff at 14:00 on a single day; another is a gradual 3-week slide. Assuming both are real, what does each shape tell you about the likely cause, and how does it change where you look first?
A sharp cliff at a timestamp points to a discrete event — a deploy, a pricing flip, a tracking break — so align it against change logs at that minute. A slow slide points to a gradual force — seasonality, cohort decay, data loss — so compare cohorts over weeks.
Common mistakes
- ✗Treating shape as irrelevant to the likely cause
- ✗Equating a cliff with an artifact and a slide with a real decline
- ✗Aligning a gradual slide to a single deploy minute
Follow-up questions
- →What shape would a slow-rolling release create?
- →How would a partial outage that self-heals appear?
MiddleDebuggingCommonA dashboard shows yesterday's orders down 90%. Find and fix the query bug behind the fake drop.
A dashboard shows yesterday's orders down 90%. Find and fix the query bug behind the fake drop.
The drop is a data-completeness artifact — yesterday's partition is still loading, so COUNT sees only the rows landed so far. Confirm it in seconds via load status. Fix the tile to drop any date not marked complete: join _load_status and filter on is_complete.
Common mistakes
- ✗Treating an incomplete partition as a real collapse
- ✗Widening the window instead of guarding completeness
- ✗Blaming dedup rather than data freshness
Follow-up questions
- →What one check confirms partition completeness in under a minute?
- →How would you stop this from paging the team next time?
MiddleDebuggingCommonAn analyst's pre/post query claims a release caused a 20% DAU drop. Find and fix its causal flaw.
An analyst's pre/post query claims a release caused a 20% DAU drop. Find and fix its causal flaw.
The pre/post split confounds the release with everything else that moved that week — the holiday and the marketing burst — so a lower 'after' number is correlation, not cause. Since rollout is gradual by app version, compare updated versus not-yet-updated users on the same days.
Open full question →Common mistakes
- ✗Reading a pre/post gap as causation despite coincident changes
- ✗Fixing the window size instead of adding a counterfactual group
- ✗Slicing the same pre/post comparison instead of comparing cohorts
Follow-up questions
- →How would a staged rollout let you estimate the release effect directly?
- →What would you check if all users updated on the same day?
MiddleDesignCommonRevenue fell 15% last month but order count is flat. Decompose revenue into its multiplicative drivers, and explain how you find which driver moved and where.
Revenue fell 15% last month but order count is flat. Decompose revenue into its multiplicative drivers, and explain how you find which driver moved and where.
Revenue = orders × average order value, and AOV = items per order × price. With orders flat, the drop sits in AOV — so decompose it: did basket size shrink or price fall? Then slice that driver by category and segment. Decomposition names the factor; slicing finds where.
Common mistakes
- ✗Assuming revenue and orders must move together
- ✗Correlating against events instead of decomposing the metric
- ✗Chasing acquisition when order count is flat
Follow-up questions
- →How would a category mix shift show up in this decomposition?
- →What if price rose but basket size fell — how do you net them?
MiddleDesignCommonA metric drops every Saturday and recovers Monday. Some call it an incident each weekend. How do you set a baseline that flags real problems without crying wolf on the weekly pattern?
A metric drops every Saturday and recovers Monday. Some call it an incident each weekend. How do you set a baseline that flags real problems without crying wolf on the weekly pattern?
Compare like with like: baseline against the same weekday, not the day before, so seasonality cancels. Model the level from prior same-weekdays and alert only when the deviation exceeds normal variation and persists past one reading. That flags a Saturday abnormal for a Saturday.
Common mistakes
- ✗Baselining against the prior day instead of the same weekday
- ✗Using a fixed percentage band across every weekday
- ✗Dropping weekend days rather than modeling the cycle
Follow-up questions
- →How would you handle holidays that shift the weekly pattern?
- →How long should a breach persist before you alert?
MiddleDesignCommonAfter slicing, the entire 20% drop is concentrated on Android 12 in one country and nowhere else. What do you conclude, what do you do next, and what do you still not know from this fact alone?
After slicing, the entire 20% drop is concentrated on Android 12 in one country and nowhere else. What do you conclude, what do you do next, and what do you still not know from this fact alone?
Concentration in one platform-country cell localizes the cause but does not name it — it fits a crash, a broken SDK, a regional outage, or a payment issue. Next check crash rates, version adoption, and logging health. You still cannot tell a real loss from a tracking break.
Common mistakes
- ✗Jumping from a localized cell straight to a confirmed code bug
- ✗Dismissing a concentrated segment signal as noise
- ✗Assuming a localized cause cannot be a tracking break
Follow-up questions
- →What single check best separates a crash from an SDK break here?
- →How would you confirm the users are gone, not just untracked?
MiddleDesignCommonConversion is down 5% right after marketing doubled top-of-funnel spend. Someone claims a product regression; you suspect a traffic-quality mix shift. How do you prove which it is?
Conversion is down 5% right after marketing doubled top-of-funnel spend. Someone claims a product regression; you suspect a traffic-quality mix shift. How do you prove which it is?
Split conversion by channel and new-vs-returning, then compare within each. A mix shift keeps per-channel rates flat while the blend tilts to lower-intent traffic; a regression drops rates inside cohorts. Recompute on the pre-spend mix — if it returns to baseline, the mix moved.
Common mistakes
- ✗Reading timing coincidence as proof it is the campaign
- ✗Assuming more spend should always lift conversion
- ✗Equal-weighting channels instead of holding the real mix
Follow-up questions
- →How would you quantify the share of the drop due to mix?
- →What within-cohort result would flip you to a product cause?
SeniorDesignCommonA metric rose 25% overnight and everyone is celebrating. Why is the investigation identical to a drop, and what does an unexplained overnight spike usually turn out to be?
A metric rose 25% overnight and everyone is celebrating. Why is the investigation identical to a drop, and what does an unexplained overnight spike usually turn out to be?
A spike is an anomaly like a drop, so run the same checks: rule out an artifact first — double-counted events, a duplicated pipeline run, a definition change, bots — then localize by segment. An unexplained overnight jump is usually not growth but instrumentation: duplicate logging or bots.
Common mistakes
- ✗Auditing drops but trusting spikes uncritically
- ✗Assuming artifacts can only suppress, never inflate
- ✗Attributing a spike to the latest feature before checks
Follow-up questions
- →What duplication bug most often inflates an event count?
- →How would bot traffic show up in your segment cuts?
SeniorDebuggingOccasionalAn anomaly alert fired 40 times last month, 38 of them noise. Redesign the detection rule so the team trusts it again.
An anomaly alert fired 40 times last month, 38 of them noise. Redesign the detection rule so the team trusts it again.
Replace the single-point threshold with a seasonal, robust, persistent rule: baseline against prior same-weekdays, not one point from 7 days ago. Flag only when the deviation clears a robust spread of a few MAD, persists past one interval, and beats a minimum absolute effect.
Open full question →Common mistakes
- ✗Loosening the threshold instead of adding seasonality and persistence
- ✗Comparing to a single reference point rather than a distribution
- ✗Firing on one reading with no minimum absolute effect
Follow-up questions
- →How would you tune the persistence and minimum-effect thresholds?
- →How do you keep the rule from missing a slow real decline?
SeniorDesignOccasionalA metric is genuinely down, present in every segment, and nothing shipped. Leadership wants to blame a competitor or a holiday. How do you attribute it to an external factor without hand-waving?
A metric is genuinely down, present in every segment, and nothing shipped. Leadership wants to blame a competitor or a holiday. How do you attribute it to an external factor without hand-waving?
Treat the external claim as a hypothesis needing evidence, not a default. First exhaust internal causes and confirm the all-segment drop is real. Then require it to predict the timing, segments, and magnitude. Use an unaffected control, and attribute only if it held while the exposed group moved.
Common mistakes
- ✗Defaulting to an external cause by elimination
- ✗Accepting temporal proximity as attribution
- ✗Skipping an unaffected control comparison
Follow-up questions
- →What makes a good control region or comparison metric here?
- →How would you size the external effect once attributed?
