Hypothesis Testing
Hypotheses, power, test choice, multiple comparisons, and Bayesian versus frequentist readouts.
12 questions
JuniorTheoryVery commonHow do you state H₀ and H₁, and why can a test only reject H₀ and never prove it?
How do you state H₀ and H₁, and why can a test only reject H₀ and never prove it?
H₀ is the no-effect, status-quo claim; H₁ is the effect you suspect. A test assumes H₀ and asks how surprising the data is: surprising data rejects H₀, otherwise you only fail to reject it — which is not proof H₀ is true.
Common mistakes
- ✗Swapping H₀ and H₁ so the effect becomes the null
- ✗Reading a non-significant result as proof of no effect
- ✗Claiming a test proves H₀ rather than failing to reject it
Follow-up questions
- →What does 'fail to reject H₀' actually let you report to a product team?
- →Why is a non-significant result not the same as evidence of no effect?
JuniorTheoryVery commonA change gives a statistically significant +0.05% lift — why might you still not ship it?
A change gives a statistically significant +0.05% lift — why might you still not ship it?
Statistical significance only means the effect is probably not zero, not how big it is. A real +0.05% lift can be too small to justify the build cost, complexity, and guardrail risk. Practical significance asks whether it clears a business threshold worth acting on.
Common mistakes
- ✗Treating any significant result as automatically worth shipping
- ✗Assuming a larger sample will grow the measured effect itself
- ✗Equating a smaller p-value with a larger, more useful effect
Follow-up questions
- →How would you set a minimum practically-significant effect before running an experiment?
- →Why can a very large sample make a trivial effect statistically significant?
JuniorTheoryVery commonWhen is a one-tailed test legitimate, and why is it usually a red flag in experiments?
When is a one-tailed test legitimate, and why is it usually a red flag in experiments?
A one-tailed test puts all of alpha on one side, detecting an effect only in a pre-declared direction. It is fine when the other direction is truly irrelevant, but a red flag because people switch to it after seeing data to halve the p-value.
Common mistakes
- ✗Choosing the tail after seeing which way the metric moved
- ✗Believing a one-tailed test is simply more powerful at no cost
- ✗Thinking a one-tailed test checks both directions at once
Follow-up questions
- →How does moving to a one-tailed test change the p-value for the same data?
- →What kind of pre-registration would make a one-tailed test defensible?
JuniorTheoryCommonWhich test fits a conversion rate, which fits an average order value, and which fits a categorical breakdown?
Which test fits a conversion rate, which fits an average order value, and which fits a categorical breakdown?
Match the test to the metric's data type. A conversion rate is a proportion → a two-proportion z-test (or chi-square form). An average order value is a mean → a two-sample t-test. Category counts versus expectation → a chi-square test of independence.
Common mistakes
- ✗Using a t-test on 0/1 conversion flags instead of a proportion test
- ✗Treating money amounts as categorical and reaching for chi-square
- ✗Picking the test by correlation strength rather than data type
Follow-up questions
- →When does the two-proportion z-test agree with a 2×2 chi-square test?
- →Why does a continuous metric like revenue call for a t-test, not a proportion test?
MiddleTheoryCommonBonferroni vs Benjamini-Hochberg — family-wise error rate versus false-discovery rate, and which fits a 40-metric dashboard?
Bonferroni vs Benjamini-Hochberg — family-wise error rate versus false-discovery rate, and which fits a 40-metric dashboard?
Bonferroni controls the family-wise error rate — any false positive — by testing each hypothesis at alpha/m, so it loses power as m grows. Benjamini-Hochberg controls the false-discovery rate — the share of false positives among rejections — and is less strict, fitting a 40-metric dashboard better.
Common mistakes
- ✗Swapping which method controls FWER versus FDR
- ✗Thinking both just divide alpha and are equivalent
- ✗Believing FDR control is stricter than FWER control
Follow-up questions
- →How does the Benjamini-Hochberg step-up procedure decide which p-values to reject?
- →When would you still prefer Bonferroni over BH on a dashboard?
MiddleTheoryCommonYou test 20 metrics and one comes back p = 0.03 — what is the chance it is noise, and what do you do?
You test 20 metrics and one comes back p = 0.03 — what is the chance it is noise, and what do you do?
Across 20 independent nulls the chance of ≥1 false positive is 1 − 0.95²⁰ ≈ 64%, so a lone p = 0.03 is probably noise. Correct for multiplicity with Bonferroni or Benjamini-Hochberg, or pre-declare one primary metric and treat the rest as exploratory.
Common mistakes
- ✗Reading the single p-value as the 3% chance it is noise
- ✗Thinking more metrics tested makes each result more reliable
- ✗Averaging the p-values to judge the whole experiment
Follow-up questions
- →How does the 64% figure change if the 20 metrics are correlated?
- →What is the difference between a primary and an exploratory metric here?
MiddleTheoryCommonAt n = 10 million almost everything is significant — what does p < 0.05 tell you then, and what instead?
At n = 10 million almost everything is significant — what does p < 0.05 tell you then, and what instead?
At huge n the standard error shrinks, so even a microscopic difference yields p < 0.05. Significance then only confirms the effect is not exactly zero — not that it matters. Look at the effect size and confidence interval against a practical threshold.
Common mistakes
- ✗Reading a tiny p-value as evidence of a large effect
- ✗Trusting significance more at large n without checking effect size
- ✗Only raising the alpha cutoff and ignoring the interval
Follow-up questions
- →Why does a confidence interval stay informative when the p-value stops being?
- →How would a minimum detectable effect stop you from chasing trivial lifts?
MiddleTheoryCommonStudent's vs Welch's t-test — how do they differ on variances, and why is Welch the safer default?
Student's vs Welch's t-test — how do they differ on variances, and why is Welch the safer default?
Student's t-test assumes the groups share a variance and pools them; Welch's uses each group's own variance and adjusts the degrees of freedom. With unequal variances or sizes, Student's inflates the Type I error while Welch's stays near nominal, so default to it.
Common mistakes
- ✗Reversing which test assumes equal variances
- ✗Thinking the difference is about normality, not variance
- ✗Believing Welch's compares medians rather than means
Follow-up questions
- →How does unequal group size interact with unequal variance to distort Student's test?
- →Why does Welch's test use fractional degrees of freedom?
SeniorTheoryOccasionalA Bayesian readout says P(B > A) = 93% while a frequentist p-value is 0.08 — what does each actually license you to claim?
A Bayesian readout says P(B > A) = 93% while a frequentist p-value is 0.08 — what does each actually license you to claim?
The Bayesian 93% is a direct probability about the parameter given model and prior, so you may act on that stated risk. The frequentist p = 0.08 is P(data this extreme | H₀); missing 0.05, you fail to reject H₀ and cannot claim an effect — it is not P(H₁).
Common mistakes
- ✗Treating 1 minus the p-value as P(the effect is real)
- ✗Reading the p-value as the probability the null is true
- ✗Calling the Bayesian posterior merely subjective and unusable
Follow-up questions
- →Why can a Bayesian posterior and a p-value disagree on the very same data?
- →How does the choice of prior change what P(B > A) = 93% actually means?
SeniorTheoryOccasionalThe rank test Mann-Whitney U on a heavy-tailed revenue metric — what does it actually test, and why is 'the median moved' wrong?
The rank test Mann-Whitney U on a heavy-tailed revenue metric — what does it actually test, and why is 'the median moved' wrong?
Mann-Whitney U tests stochastic dominance — whether a random draw from one group tends to exceed one from the other. It is not a median test: it can be significant with equal medians, or miss a real shift when distributions cross. It says which group is usually larger, not by how much.
Common mistakes
- ✗Reporting Mann-Whitney U as a test of the median difference
- ✗Treating it as a t-test on means with outliers removed
- ✗Assuming it needs normal, equal-variance distributions
Follow-up questions
- →When can Mann-Whitney U be significant even though the two medians are identical?
- →Why does a shift interpretation require assuming the two distributions share a shape?
SeniorTheoryOccasionalARPU (average revenue per user) is a ratio metric — why does a naive t-test understate its variance, and what fixes it?
ARPU (average revenue per user) is a ratio metric — why does a naive t-test understate its variance, and what fixes it?
ARPU is a ratio of two correlated random quantities, X̄/Ȳ. A naive per-row t-test treats the denominator as fixed and rows as independent, ignoring its randomness and within-user correlation, so it understates the variance and overstates significance. The delta method linearizes X̄/Ȳ to recover the true variance.
Common mistakes
- ✗Treating the user-count denominator as fixed rather than random
- ✗Thinking the correction only matters at small sample sizes
- ✗Believing the naive test overstates rather than understates variance
Follow-up questions
- →How does the analysis unit differing from the randomization unit create this problem?
- →When would you reach for a bootstrap instead of the delta method?
SeniorTheoryOccasionalDoes the t-test need normal data or normal sample means — when does the central limit theorem rescue you, and when not?
Does the t-test need normal data or normal sample means — when does the central limit theorem rescue you, and when not?
The t-test needs the mean's sampling distribution to be about normal, not the raw data. By the central limit theorem, large n makes the mean approximately normal even for skewed data. It fails at small n, or data so heavy-tailed the mean converges slowly — use a transform, rank test, or bootstrap.
Common mistakes
- ✗Requiring the raw data itself to be normal at any sample size
- ✗Assuming the CLT makes the t-test valid at any sample size
- ✗Thinking the CLT normalizes the data rather than the mean
Follow-up questions
- →How large a sample does a lognormal revenue metric need for the mean to look normal?
- →Why does a rank-based test sidestep the normality question entirely?
