A/B Testing
An A/B test is a statistical hypothesis test worn over a product decision: split users into two groups, show a different variant, and decide whether the metric difference is real or random noise. It all rests on the machinery of hypothesis testing — the null and alternative, the p-value, the significance level, power.
The traps here are conceptual, not arithmetic, and those are exactly what interviews probe: p-value read as "the probability the hypothesis is true"; the test stopped the moment p drops below 0.05; a proportion compared with a t-test on aggregates. Below are the nine notions that assemble a correct experiment: from defining errors and the p-value to sizing the sample and the trade-offs of organizing a test.
Topic map
- Type I and Type II errors — a false positive (alpha) versus a missed effect (beta), and the trade-off between them.
- p-value — the probability of the data given a true null, not the probability of the hypothesis itself.
- Statistical power — the chance of catching a real effect, equal to
1 - beta, and what raises it. - Confidence interval — a range for the true parameter and its frequentist meaning.
- Proportion criterion — the z-test and chi-square for "clicked / did not click" metrics.
- Choosing the statistical criterion — matching the test to the metric type: proportion, mean, ratio.
- Minimum detectable effect (MDE) — the smallest effect a test can resolve.
- A/B test design — computing sample size and duration without peeking.
- Experiment trade-offs — organization, cost, alternatives, and trust in fragile results.
Common traps
| Mistake | Consequence |
|---|---|
| Reading the p-value as the probability the null is true | A total misunderstanding of the test's logic — the gravest fail |
Stopping the test at the first p < 0.05 (peeking) | An inflated false-positive rate, a fragile "effect" |
| Comparing proportions (CTR) with a t-test on aggregates | Wrong criterion — you need per-user binary data and a z-test/chi-square |
| Confusing "contains 0" with "significant" for a confidence interval | An interval that does NOT contain 0 is significant — not the reverse |
| Treating the MDE as the observed effect or a ceiling | The MDE is a sensitivity floor set at design time, decided in advance |
| Ignoring variance when sizing the sample | A noisy metric needs many times more n for the same MDE |
Interview relevance
The A/B test is a favourite of product rounds because it checks statistics and product thinking at once. A candidate who distinguishes "the data given the null" from "the probability of the null", talks through the alpha–beta–power–sample link, and picks the criterion for the metric type instantly looks like someone you can trust with a production experiment.
Typical checks:
- The exact definition of a p-value and what it is not.
- Type I and Type II errors, their rates (alpha, beta), and the link to power.
- How sample size depends on alpha, power, variance, and the MDE.
- Choosing the criterion: a proportion by z-test/chi-square, a mean by t-test.
- Why a test must not be stopped early and what peeking is.
Common wrong answer: "a p-value of 0.03 means a 97% chance the effect is real". In fact a p-value is the probability of seeing data this extreme or more given that the null hypothesis is true, not the probability of the hypothesis itself.