Experimentation & Causal
An experiment is the gold standard for a causal claim. We split users into groups at random so they differ only by the treatment, show different variants, and read the metric. As long as randomization is honest, confounders balance out on average, and the gap between arms is the causal effect — not a correlation. The whole difficulty is that "honest randomization" breaks in a dozen ways in practice, and most interview questions are precisely about those ways.
First comes design: what you randomize, which metric you fix in advance, whether you have the power, and how SRM or interference silently zero out the result. Then hypothesis testing: what a p-value means and what it does not, what peeking and multiple comparisons cost you. Then causation without an A/B: when you cannot randomize, DiD, IV, RDD, and matching step in. And underneath all of it sits probability, which interviews test head-on with puzzles.
Topic map
- Experiment design — the randomization unit, the hypothesis and OEC/guardrail metrics, power/MDE/sample size, duration and weekly seasonality, SRM and A/A as validity checks, interference and novelty effects.
- Hypothesis testing — null and alternative, p-value, Type I/II errors, confidence intervals, one- vs two-tailed, multiple comparisons (Bonferroni vs Benjamini-Hochberg), test selection, statistical vs practical significance.
- Causal inference — why correlation ≠ causation, confounders, potential outcomes, DiD, instrumental variables, regression discontinuity, propensity-score matching, CUPED, and Simpson's paradox.
- Probability puzzles — conditional probability and Bayes, expected value, classic brain-teasers (birthdays, Monty Hall, coins/dice), combinatorics, and distributions.
Common traps
| Mistake | Consequence | |
|---|---|---|
| Powering on user count when you randomized clusters | Inflated significance: effective n is the number of clusters | |
| Stopping the test the moment p drops below 0.05 (peeking) | Blown Type I error — the "win" is really noise | |
| Reading a p-value as "the probability the hypothesis is true" | Wrong conclusion; a p-value is `P(data \ | H₀)` |
| Waving off an SRM of 50.8/49.2 as "a tiny deviation" | Broken assignment → the whole readout is void | |
| Declaring "no difference" off an underpowered test | Absence of evidence ≠ evidence of absence | |
| Confirming causation with a correlation and no randomization | Selection and confounders manufacture an effect that isn't there |
Interview relevance
This topic is asked to separate someone who can run and read experiments from someone who memorized the words "A/B test" and "p-value". A strong candidate names the randomization unit immediately, spots SRM, tells practical from statistical significance, and knows what to do when randomization is impossible.
Typical checks:
- The three properties of a trustworthy A/B: randomization, isolation, and one metric fixed up front.
- What a p-value actually means and why "p > 0.05" is not "no difference".
- How multiple comparisons inflate false positives and how to control them.
- How to measure an effect without an A/B: DiD with a parallel-trends check, IV, RDD, synthetic control.
Common wrong answer: "users of feature X retain 3× better, so push X to everyone." In reality that is selection: X's users were already more engaged, and on a full rollout the lift can vanish — only a randomized launch buys you causation.