Causal Inference
Measuring causation without an A/B — DiD, synthetic control, matching, IV, RDD, and the evidence hierarchy.
10 questions
JuniorTheoryVery commonCorrelation is not causation — name three distinct ways two variables can correlate with no causal link between them.
Correlation is not causation — name three distinct ways two variables can correlate with no causal link between them.
Three routes with no X→Y link. A shared common cause (a confounder) drives both. Reverse causation — the presumed effect actually drives the presumed cause. Or selection and coincidence — a collider you conditioned on, or a chance alignment in a small sample. None means X moves Y.
Common mistakes
- ✗Treating a high correlation coefficient as proof of a causal effect
- ✗Forgetting that reverse causation produces the same correlation
- ✗Ignoring that conditioning on a collider manufactures association
Follow-up questions
- →How does conditioning on a collider create a spurious correlation?
- →Which of the three routes does a randomized experiment rule out?
MiddleTheoryVery commonRank the causal-evidence hierarchy from an RCT down to raw correlation — what does each rung buy you?
Rank the causal-evidence hierarchy from an RCT down to raw correlation — what does each rung buy you?
Each rung buys more control of confounding. An RCT randomizes — every confounder balanced in expectation. A natural experiment borrows an as-if-random shock. Diff-in-diff removes fixed gaps under parallel trends. Matching balances only measured covariates. Raw correlation controls nothing.
Common mistakes
- ✗Treating matching as equivalent to randomization
- ✗Assuming a large sample lifts raw correlation to causal proof
- ✗Confusing the hierarchy of bias control with statistical power
Follow-up questions
- →Why does randomization balance unobserved confounders that matching cannot?
- →What as-if-random source turns an observational study into a natural experiment?
MiddleDesignCommonYour company launched a new pricing page in one city (Kazan) and nowhere else. You want to measure its effect on weekly revenue per user with a difference-in-differences design — comparing the before-vs-after change in Kazan against the same change in a set of untreated control cities. A colleague asks what assumption makes this valid and how you would defend it. Which answer correctly states the parallel-trends assumption and a sound way to check it?
Your company launched a new pricing page in one city (Kazan) and nowhere else. You want to measure its effect on weekly revenue per user with a difference-in-differences design — comparing the before-vs-after change in Kazan against the same change in a set of untreated control cities. A colleague asks what assumption makes this valid and how you would defend it. Which answer correctly states the parallel-trends assumption and a sound way to check it?
Parallel trends — absent the launch, Kazan and the controls would have moved on the same trend, parallel, not at equal levels. You cannot prove it, only support it — plot several pre-launch periods and confirm the two groups were already trending in parallel before treatment began.
Common mistakes
- ✗Requiring equal pre-period levels instead of equal trends
- ✗Believing parallel trends can be proven rather than only supported
- ✗Checking the post-treatment period instead of the pre-period
Follow-up questions
- →What breaks diff-in-diff if a shock hits only the control cities mid-study?
- →How would an event-study plot make a pre-trend violation visible?
MiddleTheoryCommonOpt-in users retain better. Why is the naive comparison worthless, and what does the method propensity-score matching fix?
Opt-in users retain better. Why is the naive comparison worthless, and what does the method propensity-score matching fix?
Opt-in is self-selection: users who opt in were already more engaged, so the gap confounds the feature's effect with who chose it. Propensity-score matching pairs opted-in and non-users with equal modelled probability of opting in, balancing observed covariates so like is compared with like.
Common mistakes
- ✗Reading the opted-in retention gap as the feature's causal effect
- ✗Assuming matching also balances unobserved confounders
- ✗Matching on the outcome instead of the propensity to opt in
Follow-up questions
- →What unobserved confounder could survive propensity-score matching here?
- →How do you check covariate balance after matching on the score?
SeniorDesignCommonYou must measure the sales lift of a TV and outdoor campaign, but you cannot randomize individual users — everyone in a region sees the same ads. You decide on a geo-experiment: randomize whole regions into test and control, run the campaign only in test regions, and compare. A colleague asks how you assign regions, how you read the result given only a few large units, and what threatens validity. Which answer describes a sound geo-experiment design and readout?
You must measure the sales lift of a TV and outdoor campaign, but you cannot randomize individual users — everyone in a region sees the same ads. You decide on a geo-experiment: randomize whole regions into test and control, run the campaign only in test regions, and compare. A colleague asks how you assign regions, how you read the result given only a few large units, and what threatens validity. Which answer describes a sound geo-experiment design and readout?
Randomize matched regions into test and control, pairing on pre-period sales and trend; run ads only in test. With few, large units, read the lift via diff-in-diff or synthetic control on aggregated sales, inference from permutation — not user-level tests. Border spillover and too few regions threaten it.
Common mistakes
- ✗Running a user-level test on a few aggregated regional units
- ✗Selecting test regions by size instead of randomizing matched pairs
- ✗Ignoring ad spillover across neighboring region borders
Follow-up questions
- →Why does treating users as the unit understate variance in a geo test?
- →How does border spillover bias the measured lift, and how do you limit it?
SeniorTheoryOccasional'Users of feature X retain 3× better, so let us push X to everyone.' Demolish this causal claim in one minute.
'Users of feature X retain 3× better, so let us push X to everyone.' Demolish this causal claim in one minute.
It confuses correlation with causation. X's users self-select — already more engaged, so they'd retain better without X too. The 3× mixes in selection, reverse causation (loyal users seek X out), and confounders. Rolled out to all, the lift can vanish; only a randomized rollout isolates the effect.
Common mistakes
- ✗Reading the 3× retention gap as X's causal effect
- ✗Ignoring self-selection of engaged users into X
- ✗Assuming one covariate adjustment removes all confounding
Follow-up questions
- →What single experiment would cleanly measure X's causal effect on retention?
- →How could reverse causation alone produce the observed 3× gap?
SeniorDesignOccasionalYour loyalty tier flips from Silver to Gold the instant a customer's cumulative spend crosses 10,000 ₽, and Gold unlocks free shipping. You want the causal effect of Gold status on repeat-purchase rate. You cannot randomize, but the sharp 10,000 ₽ threshold is exploitable with a regression-discontinuity design — comparing customers just above the cutoff with those just below. A colleague asks how the design identifies the effect and what could invalidate it. Which answer is correct?
Your loyalty tier flips from Silver to Gold the instant a customer's cumulative spend crosses 10,000 ₽, and Gold unlocks free shipping. You want the causal effect of Gold status on repeat-purchase rate. You cannot randomize, but the sharp 10,000 ₽ threshold is exploitable with a regression-discontinuity design — comparing customers just above the cutoff with those just below. A colleague asks how the design identifies the effect and what could invalidate it. Which answer is correct?
Right at the cutoff, landing just above or below 10,000 ₽ is as good as random, so customers on either side are comparable and the jump in repeat rate at the threshold estimates Gold's local effect. It breaks if customers manipulate spend to cross the line, or if something else also changes at exactly 10,000 ₽.
Common mistakes
- ✗Comparing all Gold to all Silver instead of customers near the cutoff
- ✗Ignoring manipulation of spend to cross the threshold
- ✗Extrapolating the local effect at the cutoff to the whole base
Follow-up questions
- →How does a density test around the cutoff reveal running-variable manipulation?
- →Why is a regression-discontinuity effect local to the threshold, not global?
SeniorDesignOccasionalOnly one city (Kazan) received the change, and no single control city is a good match — each differs on size, seasonality, or category mix. You decide to build a synthetic control: a weighted blend of untreated 'donor' cities chosen so the blend tracks Kazan's pre-launch outcome closely. A stakeholder asks how you construct it and, crucially, how you know it is trustworthy before you ever read the effect. Which answer describes a sound synthetic-control build and its pre-period validation?
Only one city (Kazan) received the change, and no single control city is a good match — each differs on size, seasonality, or category mix. You decide to build a synthetic control: a weighted blend of untreated 'donor' cities chosen so the blend tracks Kazan's pre-launch outcome closely. A stakeholder asks how you construct it and, crucially, how you know it is trustworthy before you ever read the effect. Which answer describes a sound synthetic-control build and its pre-period validation?
Choose donor-city weights so the synthetic blend matches Kazan's pre-launch trajectory and key predictors, fit on pre-period data only. Validate before reading any effect — the synthetic must track real Kazan tightly across the whole pre-window, and placebo runs on untreated donors confirm the gap is not chance.
Common mistakes
- ✗Fitting the weights on data that includes the post-launch period
- ✗Skipping the pre-period fit check before reading the effect
- ✗Ignoring placebo tests that would reveal a spurious gap
Follow-up questions
- →How does a placebo test on untreated donors calibrate the effect's significance?
- →Why can a poor pre-period fit invalidate the whole synthetic-control estimate?
SeniorDebuggingRareA difference-in-differences readout shows a big effect, but the pre-period trends were already diverging — diagnose it.
A difference-in-differences readout shows a big effect, but the pre-period trends were already diverging — diagnose it.
Parallel trends is violated — treatment was already diverging upward from control before week 0. Diff-in-diff charges that pre-existing gap to the launch, so most of the 'effect' is bias, not impact. Report that the estimate is not credible — you need a control with matching pre-trends before claiming any effect.
Open full question →Common mistakes
- ✗Reading a diverging pre-trend as treatment momentum
- ✗Believing more control units fix a parallel-trends violation
- ✗Reporting the diff-in-diff estimate despite a failed pre-trend check
Follow-up questions
- →What method could still recover an effect when pre-trends are not parallel?
- →How would an event-study plot quantify the pre-trend violation?
SeniorTheoryRareMatching balances only what you measured. Explain unobserved confounding, and how an instrumental variable gets around it.
Matching balances only what you measured. Explain unobserved confounding, and how an instrumental variable gets around it.
Matching balances only observed covariates, so an unobserved confounder — say motivation — still biases it. An instrumental variable shifts treatment but touches the outcome only through treatment, and is independent of the confounder. Its as-if-random variation identifies the effect the confounder would hide.
Common mistakes
- ✗Assuming matching removes unobserved confounders too
- ✗Adding the instrument as an ordinary regression control
- ✗Choosing an instrument that affects the outcome directly, violating exclusion
Follow-up questions
- →What are the exclusion restriction and relevance conditions for a valid instrument?
- →Why does a weak instrument inflate the variance of the estimate?
