Experiment Design
Beyond a vanilla A/B — SRM, CUPED, sequential and switchback designs, clusters, interference, and ramps.
16 questions
JuniorTheoryVery commonWhat three properties make an online A/B test result trustworthy?
What three properties make an online A/B test result trustworthy?
Three things: random assignment, so arms differ only by the treatment and confounders balance out; isolation, so users in one arm don't affect the other; and one primary metric fixed before launch, so you can't pick a winner after seeing the data.
Common mistakes
- ✗Thinking a large enough sample removes the need for randomization
- ✗Assuming arms stay isolated even when treated users interact with control users
- ✗Choosing the metric after seeing which one moved
Follow-up questions
- →How does randomization balance confounders you never even measured?
- →What breaks when users in one arm influence the other arm?
MiddleTheoryVery commonWhy does an underpowered A/B test that 'shows no difference' prove nothing?
Why does an underpowered A/B test that 'shows no difference' prove nothing?
Power is the probability of detecting a true effect; underpowered, a test has a high false-negative rate, so a non-significant result is expected even when a real effect exists — absence of evidence, not evidence of absence. 'p > 0.05' is not 'no difference': the interval is too wide to exclude effects that matter.
Common mistakes
- ✗Reading 'p > 0.05' as proof the arms are equal
- ✗Ignoring the confidence-interval width when interpreting a null
- ✗Confusing precision (power) with bias in the point estimate
Follow-up questions
- →What would you compute to decide if a null result is informative?
- →How does an equivalence test differ from failing a superiority test?
MiddleDesignCommonYou roll a change out 1% → 5% → 25% → 50%, holding at each stage. You have real-time operational guardrails (latency, error rate, crash rate) and slower business metrics (conversion, revenue per user). Describe what you monitor at each ramp step and what triggers an automatic rollback, and explain why you ramp at all instead of going straight to a 50/50 test. Be specific about which class of signal gates each stage and how the statistics change as the exposed population grows.
You roll a change out 1% → 5% → 25% → 50%, holding at each stage. You have real-time operational guardrails (latency, error rate, crash rate) and slower business metrics (conversion, revenue per user). Describe what you monitor at each ramp step and what triggers an automatic rollback, and explain why you ramp at all instead of going straight to a 50/50 test. Be specific about which class of signal gates each stage and how the statistics change as the exposed population grows.
You ramp to limit blast radius. At 1% a catastrophic bug — crashes, 5xx, latency spikes — hits few users and trips fast operational guardrails; these gate the early stages. Widening to 25–50% gives the exposure to read slower business metrics at power, gating later stages. Rollback fires automatically on any guardrail breach past a threshold.
Common mistakes
- ✗Reading business metrics at 1% as if they were already powered
- ✗Letting slow business significance, not fast guardrails, gate early stages
- ✗Claiming a straight 50/50 launch is safer than a ramp
Follow-up questions
- →Why can't you trust the conversion metric at the 1% stage?
- →What guardrail thresholds would you wire to auto-rollback?
MiddleDesignCommonYour team ships dozens of small ranking and UI wins each quarter, each individually A/B-tested as positive, yet the north-star metric is only drifting up slowly. Leadership asks whether the wins actually compound or the metric is rising on its own. You propose a 1% permanent holdout: at the start of the year 1% of users are randomly assigned and kept on the old baseline, never exposed to any of the year's launches. Explain what this holdout measures that the individual A/B tests cannot, and what it costs — for the held-out users and for the business. Say how you would size it and how you would read it out at year end.
Your team ships dozens of small ranking and UI wins each quarter, each individually A/B-tested as positive, yet the north-star metric is only drifting up slowly. Leadership asks whether the wins actually compound or the metric is rising on its own. You propose a 1% permanent holdout: at the start of the year 1% of users are randomly assigned and kept on the old baseline, never exposed to any of the year's launches. Explain what this holdout measures that the individual A/B tests cannot, and what it costs — for the held-out users and for the business. Say how you would size it and how you would read it out at year end.
Per-launch A/B tests measure each change alone, missing interactions and drift, so summing small lifts overstates the total. A permanent holdout on last year's baseline measures the true cumulative effect of everything shipped, net of seasonality. The cost: held-out users forgo a year of gains, hence ~1%, but powered for a small effect.
Common mistakes
- ✗Assuming the yearly impact equals the sum of per-launch lifts
- ✗Treating a 1% holdout as costless because it is small
- ✗Believing a long holdout can never be contaminated
Follow-up questions
- →How would you size a holdout to detect a small aggregate lift?
- →What contaminates a long-running holdout, and how do you detect it?
MiddleDesignCommonBefore turning an experiment on, you write down and lock four things: the single primary metric, the minimum detectable effect (MDE) with the sample size and duration it implies, the guardrail metrics, and the decision rule ('ship if the primary is significantly positive and no guardrail regresses'). This is pre-registration. For each of the four pre-commitments, explain concretely which analytical abuse it prevents once the data start arriving, and what an analyst could otherwise do if it were left open. Then name one thing pre-registration does not protect against.
Before turning an experiment on, you write down and lock four things: the single primary metric, the minimum detectable effect (MDE) with the sample size and duration it implies, the guardrail metrics, and the decision rule ('ship if the primary is significantly positive and no guardrail regresses'). This is pre-registration. For each of the four pre-commitments, explain concretely which analytical abuse it prevents once the data start arriving, and what an analyst could otherwise do if it were left open. Then name one thing pre-registration does not protect against.
Locking the primary metric stops metric shopping; locking the MDE and sample size stops optional stopping (peeking, calling it at p < 0.05) that inflates false positives; locking guardrails stops ignoring a regression; locking the decision rule stops post-hoc rationalising. It cannot fix a broken experiment.
Common mistakes
- ✗Believing pre-registration neutralises SRM or interference bias
- ✗Treating the locked plan as revisable documentation, not a commitment
- ✗Thinking the sample-size lock is only about budget, not error rates
Follow-up questions
- →Why does optional stopping inflate the false-positive rate?
- →How would you handle a genuinely better metric discovered mid-experiment?
MiddleDebuggingCommonA 50/50 split logs 50.8/49.2 across 2M users. What does this SRM signal, and how do you fix the assignment below?
A 50/50 split logs 50.8/49.2 across 2M users. What does this SRM signal, and how do you fix the assignment below?
SRM means the split deviates from the intended ratio beyond chance — χ² on the arm counts gives a tiny p-value — so assignment or logging is broken and the readout is void. The bug: eligibility is tested only on the treatment branch, so ineligible users fall to control; filter them out before bucketing.
Open full question →Common mistakes
- ✗Dismissing a significant SRM as harmless rounding noise
- ✗Trying to reweight or patch the arms instead of fixing assignment and re-running
- ✗Applying an eligibility or logging filter to only one arm
Follow-up questions
- →Which χ² statistic and threshold would you use to flag the SRM?
- →Why can't you salvage the biased data by reweighting after the fact?
MiddleTheoryOccasionalMigrating infrastructure, you must prove the new system is not worse. How do the hypotheses flip in a non-inferiority test?
Migrating infrastructure, you must prove the new system is not worse. How do the hypotheses flip in a non-inferiority test?
A superiority test has H0: no difference. Non-inferiority flips it: pick a margin δ — the largest drop you tolerate — with H0: new is worse by at least δ. Declare non-inferiority when the CI for (new − control) lies entirely above −δ. Choosing δ is the crux: too loose passes regressions, too tight needs huge samples.
Common mistakes
- ✗Reading a non-significant superiority test as proof of non-inferiority
- ✗Skipping the tolerance margin δ or setting it after seeing the data
- ✗Using a two-sided interval containing zero as the criterion
Follow-up questions
- →How would you choose the margin δ for a latency migration?
- →Why must δ be fixed before the experiment, not after?
SeniorDesignOccasionalYou randomize a policy by city — some cities get treatment, others control — because the intervention affects the whole local market. A colleague powers the test on the millions of users across those cities and reports a significant win on a handful of cities per arm. Explain why the effective sample size is the number of cities, not the number of users, which statistical quantity drives this, and what it implies for how many clusters you actually need and how you must compute the standard errors.
You randomize a policy by city — some cities get treatment, others control — because the intervention affects the whole local market. A colleague powers the test on the millions of users across those cities and reports a significant win on a handful of cities per arm. Explain why the effective sample size is the number of cities, not the number of users, which statistical quantity drives this, and what it implies for how many clusters you actually need and how you must compute the standard errors.
Users in a city share local shocks — weather, marketing, one market — so they aren't independent; treatment is assigned at city level, so the city is the unit. Intra-cluster correlation inflates variance by the design effect 1+(m−1)ρ, so effective n tracks city count, not users. You need many cities and clustered standard errors, not a few clusters.
Common mistakes
- ✗Powering on user count instead of cluster count
- ✗Believing a city fixed effect restores a valid user-level test
- ✗Skipping clustered standard errors at the city level
Follow-up questions
- →How does the design effect 1+(m−1)ρ scale as cities get bigger?
- →Why don't clustered standard errors help when you have only a few clusters?
SeniorTheoryOccasionalWhen does a multi-armed bandit beat a fixed-allocation A/B, and what does adaptive allocation cost you in inference?
When does a multi-armed bandit beat a fixed-allocation A/B, and what does adaptive allocation cost you in inference?
A bandit shifts traffic toward the winning arm, minimising regret — right for optimising cumulative reward over a short horizon or many arms, not a precise effect. The cost: allocation is data-dependent, so fixed-n intervals and p-values are biased and the loser arm gives a noisy estimate. Use a fixed A/B when a lasting decision needs a clean readout.
Common mistakes
- ✗Treating a bandit as a drop-in that keeps A/B-grade inference
- ✗Believing the starved loser arm still gives a precise effect estimate
- ✗Using a bandit for a lasting decision that needs an unbiased readout
Follow-up questions
- →Why do data-dependent arm sizes bias a naive confidence interval?
- →For which decisions is minimising regret worth a noisier effect estimate?
SeniorDesignOccasionalYou test a messaging feature: a new compose flow makes treated users send more messages. But treated users message control users, who then reply — so the control arm's activity rises too. Explain why this interference biases a standard user-level A/B and in which direction, then explain how ego-cluster randomization — a network-clustering technique that assigns whole social neighbourhoods to the same arm — contains the spillover, and what it costs you in power.
You test a messaging feature: a new compose flow makes treated users send more messages. But treated users message control users, who then reply — so the control arm's activity rises too. Explain why this interference biases a standard user-level A/B and in which direction, then explain how ego-cluster randomization — a network-clustering technique that assigns whole social neighbourhoods to the same arm — contains the spillover, and what it costs you in power.
Treatment spills onto control through the social graph, so control is partly treated: the arm gap shrinks and a user-level A/B dilutes the true effect toward zero. Ego-cluster randomization puts each user's neighbourhood in one arm, so spillover stays inside the cluster. The cost is far fewer independent clusters than users — lower effective n and power.
Common mistakes
- ✗Getting the direction wrong — thinking spillover inflates rather than dilutes
- ✗Believing you can delete contaminated control users and keep full power
- ✗Assuming clustering raises the number of independent units
Follow-up questions
- →Why does spillover bias the estimate toward zero rather than away?
- →How do you define the ego-clusters when the graph is dense?
SeniorTheoryOccasionalHow do sequential tests — mSPRT or group-sequential — legitimise the early stopping that naive peeking cannot?
How do sequential tests — mSPRT or group-sequential — legitimise the early stopping that naive peeking cannot?
Peeking retests the same data at fixed α, so each look is a fresh chance to cross 0.05 and type-I climbs above α. Sequential methods pre-commit a stopping rule and budget error across looks — group-sequential via alpha-spending boundaries, mSPRT via an always-valid statistic — holding the rate at α while allowing an early stop.
Common mistakes
- ✗Believing repeated looks at a fixed α leave the type-I rate unchanged
- ✗Thinking a single stricter cutoff replaces per-look error spending
- ✗Assuming a large enough fixed sample makes free peeking valid
Follow-up questions
- →Why does each additional look inflate the false-positive rate?
- →How does an alpha-spending function differ from a fixed Bonferroni split?
SeniorDesignOccasionalYou are testing a new surge-pricing algorithm for a ride-hailing marketplace. A user-level A/B is invalid: prices and driver supply are set market-wide, so a rider bucketed to 'treatment' still lives in the equilibrium that control riders help create — the two arms leak into each other. You cannot split the marketplace cleanly by user. Explain how a switchback design — a time-based randomization technique — lets you measure the algorithm's effect here: what unit it randomizes, why that removes the leakage, and the main statistical price you pay for using it.
You are testing a new surge-pricing algorithm for a ride-hailing marketplace. A user-level A/B is invalid: prices and driver supply are set market-wide, so a rider bucketed to 'treatment' still lives in the equilibrium that control riders help create — the two arms leak into each other. You cannot split the marketplace cleanly by user. Explain how a switchback design — a time-based randomization technique — lets you measure the algorithm's effect here: what unit it randomizes, why that removes the leakage, and the main statistical price you pay for using it.
Switchback randomizes a whole market between treatment and control over time slots, so the whole equilibrium — prices, supply — is on or off each window; on-windows vs off-windows measure the market-level effect with no cross-arm leakage. The price: the unit is the time-slot, so effective n is small and windows autocorrelate, cutting power.
Common mistakes
- ✗Alternating individual users and calling it switchback
- ✗Ignoring autocorrelation between adjacent time windows
- ✗Treating the small number of time-slots as if it were the user count
Follow-up questions
- →How would you choose the length of each switchback window?
- →Why do carryover effects force buffer periods between switches?
SeniorDesignOccasionalA marketplace change — hiding seller contact info until checkout — helps buyers but may hurt sellers. Buyers and sellers transact with each other, so you cannot cleanly randomize both sides at once: a treated buyer still meets sellers who also serve control buyers. You still owe leadership a read on the net two-sided effect before launch. Describe a design that gives a defensible estimate despite the cross-side interference, why a naive user-level split is biased, and the concrete trade-off the fix forces on you.
A marketplace change — hiding seller contact info until checkout — helps buyers but may hurt sellers. Buyers and sellers transact with each other, so you cannot cleanly randomize both sides at once: a treated buyer still meets sellers who also serve control buyers. You still owe leadership a read on the net two-sided effect before launch. Describe a design that gives a defensible estimate despite the cross-side interference, why a naive user-level split is biased, and the concrete trade-off the fix forces on you.
Randomize at a level containing both sides of a transaction — market/geo or a two-sided cluster — so a treated buyer mostly meets treated sellers and spillover stays in-arm, giving a net effect. A naive user-level split leaks treatment across sides, biasing the estimate toward zero. The trade-off: far fewer independent units, so lower power and a longer test.
Common mistakes
- ✗Adding two separate one-sided A/Bs as if interference cancels
- ✗Ignoring the seller side because the change is buyer-facing
- ✗Believing reweighting removes cross-side leakage at no power cost
Follow-up questions
- →Why does a user-level split bias the two-sided estimate toward zero?
- →How would you size a geo experiment to detect a small net effect?
SeniorTheoryRareHow does the variance-reduction technique CUPED use pre-experiment data to raise power, and what makes a covariate good?
How does the variance-reduction technique CUPED use pre-experiment data to raise power, and what makes a covariate good?
CUPED subtracts θ(X−E[X]) from the metric Y, where X is a pre-experiment covariate and θ=Cov(Y,X)/Var(X); this strips the variance X explains, cutting it by 1−ρ². Because X predates assignment it can't shift the effect, only its variance. A good covariate is pre-treatment and strongly correlated with the outcome Y.
Common mistakes
- ✗Choosing a covariate measured during the experiment instead of before it
- ✗Believing CUPED changes the effect estimate, not just its variance
- ✗Thinking variance falls from a smaller sample rather than from correlation ρ
Follow-up questions
- →Why must the covariate be measured strictly before assignment?
- →How much variance reduction do you get if ρ between Y and X is 0.7?
SeniorDesignRareTwo teams launch experiments on the same checkout page in the same week: team A tests a new button colour, team B tests a new shipping-cost display. If both randomize users independently, some users land in A-treatment and B-treatment at once. Explain when this overlap creates a real problem versus when it is harmless, and how a layered (orthogonal) assignment system lets many experiments run concurrently on one surface without confounding each other. Be specific about what an interaction effect does to each readout.
Two teams launch experiments on the same checkout page in the same week: team A tests a new button colour, team B tests a new shipping-cost display. If both randomize users independently, some users land in A-treatment and B-treatment at once. Explain when this overlap creates a real problem versus when it is harmless, and how a layered (orthogonal) assignment system lets many experiments run concurrently on one surface without confounding each other. Be specific about what an interaction effect does to each readout.
Independent randomization keeps the assignments orthogonal, so each effect is estimated right on average despite the overlap — the other test just adds noise. The danger is a real interaction: if effects aren't additive, each readout is biased by the other. A layered system hashes each experiment into its own layer to keep arms orthogonal; suspected interactions share a layer.
Common mistakes
- ✗Thinking overlapping experiments must always run sequentially
- ✗Believing dropping the overlap cell keeps both tests unbiased and powered
- ✗Assuming randomization rules out interaction effects
Follow-up questions
- →Why does orthogonal assignment leave each main effect unbiased?
- →When would you deliberately put two experiments in the same layer?
