A/B Testing

A Type I error is a false positive — rejecting a true null hypothesis (seeing an effect that is not there); its rate is the significance level alpha. A Type II error is a false negative — failing to reject a false null (missing a real effect); its rate is beta, and power equals 1 - beta.

A p-value is the probability of data at least as extreme as observed, assuming the null is true. A small p-value means the data is unlikely under the null, so you reject it. It is not the probability that the null is true.

Sample size comes from alpha, power (1 - beta), the baseline metric's variance, and the MDE — n grows with variance and falls with the square of the MDE. Duration must reach that n and also cover full business cycles (at least 1-2 weeks for weekday/weekend seasonality), avoid novelty effects, and never stop early, since peeking inflates false positives.

A confidence interval is a range that, under repeated sampling, contains the true parameter (1 - alpha)% of the time — not a 95% probability the true value lies in one fixed interval. In A/B tests you build the CI of the difference between groups: if it excludes 0 the effect is significant, and its width quantifies the precision.

An A/A test runs two identical variants to validate the platform — randomization, bucketing, and metric logging. With no real difference it should read significant only about alpha of the time, so a genuinely significant A/A means a broken pipeline, not a real effect.

The MDE is the smallest true effect a test is powered to detect reliably at the chosen significance and power. It grows with metric variance and shrinks as sample size grows: for fixed alpha and power, a larger n lets you detect a smaller effect. It is set at design time to size the experiment.

CTR is a proportion — each user contributes a Bernoulli 0/1 (clicked or not). Compare the two groups with a two-proportion z-test or a chi-square test of proportions. For large n a t-test on the per-user binary indicator also works. The key is using per-user binary data, not aggregated rates, and checking n for the normal approximation.

A non-significant p-value is not proof the feature is dead — absence of evidence is not evidence of absence. The confidence interval shows the range of effects still compatible with the data. A tight interval hugging 0 with adequate power means no meaningful effect; a wide one spanning large gains means the test was underpowered, not negative.

A significant result below the MDE means the test was underpowered for that effect, so the finding is fragile: the false-negative risk was high, and a significant sub-MDE effect carries a higher chance of being a fluke or peeking artifact. Trust the significance for the decision but treat it skeptically — re-estimate variance and replicate.

Start from a hypothesis, target metric, and MDE, then design the split and a monitoring plan with guardrail metrics. Coordinate with marketing, comms, and engineering the test touches. Estimate the cost including the opportunity cost of holding back the change, and justify the test even when costly. Know the alternatives (before/after, holdout) and why A/B beats them under limited time.

Every look is another chance to cross 0.05 under the null, so daily peeking over two weeks lifts the Type I error far above the nominal 5% — often to 20-30%. The single threshold holds only for one pre-planned analysis. Instead fix the horizon up front or use a sequential alpha-spending method that holds the rate at alpha.

Cut variance with CUPED or stratification — it buys sensitivity with no extra traffic but needs a stable pre-period covariate. Analyze only the triggered population to remove dilution. Raise the MDE, accepting missed smaller wins. Run longer, paying opportunity cost. Loosen alpha or power, trading away error control.

Fix one hypothesis and a single primary metric — checkout conversion — randomize by user (not session), and power for a realistic MDE at alpha 0.05. Run the full two weeks to cover weekly seasonality, add guardrails like errors and revenue, and analyze once at the end. Refuse to peek-and-stop or re-bucket mid-test — each inflates the false-positive rate.

You cannot call them equal by eye — the difference must be tested for statistical significance, not just observed. For the two conversion rates (proportions) use a two-proportion z-test or chi-square; for the difference in average purchase frequency (means) use a t-test. Account for sample sizes and confounders like audience or device mix before concluding the funnels differ.

Split users by tenure and first-exposure date: novelty concentrates in existing users and decays as each cohort ages, while a real win holds steady in never-exposed new users. Pure novelty trends toward zero; a real effect plateaus above it. Ship on the stabilized late-window estimate, not the day-3 spike.

The win may have been a false positive, or the winner's curse inflated a sub-MDE effect that regresses to the mean. Novelty may have faded, the period may differ seasonally, or a sample-ratio mismatch skewed the arms. At 100 percent there is no holdback, and some effects appear only at full scale. Confirm with a small long-term holdout and a pipeline re-check.

This is Simpson's paradox: an uneven mix of segments across arms reverses the pooled result. If treatment drew more traffic from a high-converting platform, its pooled rate rises even while it loses in every segment. The mix gap is usually a sample-ratio mismatch, so the aggregate is untrustworthy; ship the segment-consistent direction.

Make revenue-per-user the primary metric, not CTR. CTR is a proxy a clickbait variant can inflate while hurting the outcome you care about — the surrogate trap where optimizing the proxy degrades the goal. Set guardrails on complaints, unsubscribes, and retention, with tripwires that halt the rollout on any breach.

Significance is not practical significance — on huge traffic a trivial effect clears p below 0.05 easily. If 0.3 percent sits below your MDE, the estimate is noisy and winner's-curse inflated. But tiny wins ratchet: many independent gains compound. Ship if maintenance cost is near zero and the lower bound stays positive; drop it if fragile.