An A/A test came back significant on a clean sample — selection bias, peeking, or bad luck?
An A/A test runs the identical experience in both arms — any 'winner' is pure noise. Diagnose the readout below.
Constraints: α = 0.05, a two-proportion z-test, a planned 14-day run, deterministic assignment per user.
A/A test — identical experience in both arms
Arm A: 40,102 users, conversion 4.81%
Arm B: 40,061 users, conversion 5.02%
Two-proportion z-test: p = 0.036 (significant at alpha = 0.05)
Data QA: no NULLs, split 50.05 / 49.95, no duplicate users
Call: stopped at the first p < 0.05, on day 4 of the planned 14Name the cause of the false significance and the fix.
An A/A test has no real effect, so p = 0.036 is expected — at α = 0.05, 5% of clean A/A tests hit significance by chance. The tell is peeking: stopping at the first p < 0.05 before the planned horizon inflates the false-positive rate. A balanced split rules out selection bias.
- ✗Reading any single significant A/A result as a broken pipeline
- ✗Treating repeated daily peeks as independent, harmless looks
- ✗Forgetting the 5% false-positive rate applies to A/A tests too
- →How do sequential testing methods control the peeking problem?
- →What sample-ratio check would confirm the split is truly balanced?
An A/A test compares the identical experience, so the true effect is zero and any 'difference' is noise. At α = 0.05 the test is designed to reject the null 5% of the time by chance, so a single p = 0.036 is expected and proves nothing on its own. The real problem is how the test was stopped.
Why p = 0.036 is not a signal:
- zero true effect → under H0 about 5% of tests give p < 0.05
- stopped at first p < 0.05, day 4 of 14 → repeated peeking
- repeated looks without correction → real FPR ≈ 15-30%, not 5%
Checks, in order:
- split 50.05 / 49.95 on 80k → χ² not significant → no SRM → not selection bias
- run to day 14 OR use a sequential test with alpha-spending (e.g. O'Brien-Fleming)
- replicate on fresh data: a true A/A should NOT stay significantVerdict: the cause is peeking (stopping at the first significant look), not selection bias and not a data bug. Do not call a winner; run to the horizon or apply a sequential criterion, then re-check on a fresh A/A.
