A/B Testing & Experimentation
Experiment design, peeking, multiple comparisons, novelty and network effects, switchback, CUPED, the offline–online gap, the OEC and A/A tests.
11 questions
JuniorTheoryVery commonWhat are H0 and H1 in an A/B test, what do the significance level and power control, and what does significance license?
What are H0 and H1 in an A/B test, what do the significance level and power control, and what does significance license?
H0 says variant and control have the same effect, H1 that they differ. The significance level is the false-positive rate accepted when H0 holds; power is the chance of detecting a real effect of a given size. Significance licenses rejecting H0, nothing more.
Common mistakes
- ✗Reading the p-value as the probability that H0 is true
- ✗Treating significance as proof that the measured lift size is correct
- ✗Ignoring power, so a flat result gets read as evidence of no effect
Follow-up questions
- →Why does an underpowered test make a significant win overstate the effect, the selection bias called winner curse?
- →What changes in the threshold when you run a one-sided instead of a two-sided test?
MiddleDesignVery commonYour experimentation platform has produced three surprising model wins in a row and nobody on the team trusts the splitter any more. Before the next launch you are asked to run an A/A test on the same traffic, the same assignment service and the same metric pipeline. Explain what an A/A test is and what it isolates, what result you expect from a healthy platform, what a failed A/A actually tells you, and what you would inspect inside the splitter and the metric join when it fails.
Your experimentation platform has produced three surprising model wins in a row and nobody on the team trusts the splitter any more. Before the next launch you are asked to run an A/A test on the same traffic, the same assignment service and the same metric pipeline. Explain what an A/A test is and what it isolates, what result you expect from a healthy platform, what a failed A/A actually tells you, and what you would inspect inside the splitter and the metric join when it fails.
An A/A test splits traffic between two identical variants, so any significant difference is platform error, not effect. A healthy platform fails at roughly the alpha rate. A failure means biased assignment or a broken metric join — inspect hash, bucket balance, filters.
Common mistakes
- ✗Expecting a healthy A/A test to never come back significant
- ✗Blaming sample size when assignment or the metric join is biased
- ✗Confusing an A/A test with an offline replay of the model
Follow-up questions
- →Which hashing mistake makes the two buckets correlate with a user attribute such as signup date?
- →How would you run a continuous A/A stream alongside live experiments?
MiddleDesignVery commonYou are planning the launch experiment for a new ranking model and the product team wants to know how long it must run before a decision. Take them through sizing: how you choose the single decision metric you power on, how the minimum detectable effect MDE, the significance level and the target power set the required sample per arm, how you turn that sample into calendar days from daily eligible traffic, and what you trade away when the required sample is simply unreachable inside the quarter.
You are planning the launch experiment for a new ranking model and the product team wants to know how long it must run before a decision. Take them through sizing: how you choose the single decision metric you power on, how the minimum detectable effect MDE, the significance level and the target power set the required sample per arm, how you turn that sample into calendar days from daily eligible traffic, and what you trade away when the required sample is simply unreachable inside the quarter.
Pick one decision metric; the sample per arm then follows from its variance, the MDE, alpha and power — halving the MDE costs four times the traffic. Divide by daily eligible users for duration. If unreachable, raise the MDE or cut variance, never power.
Common mistakes
- ✗Assuming the required sample scales linearly with the MDE
- ✗Choosing the MDE after seeing the observed effect
- ✗Silently accepting low power instead of renegotiating the MDE
Follow-up questions
- →How does adding a third arm change both the required traffic and the comparison count?
- →Why does a highly skewed revenue metric need more traffic than a binary conversion metric?
SeniorDesignVery commonA marketplace search team has trained a new ranking model that beats production on the offline ranking metric NDCG by 4 percent, and you own the launch experiment. Design it end to end: choose the unit of randomization and justify it against the alternatives, define the overall evaluation criterion OEC the decision rests on, name the guardrail metrics that can block a launch even when the OEC wins, pick a minimum detectable effect and derive a duration from roughly 400k daily searches by 120k daily users, and say how you would stop the model from being retrained on the data it produced during the experiment.
A marketplace search team has trained a new ranking model that beats production on the offline ranking metric NDCG by 4 percent, and you own the launch experiment. Design it end to end: choose the unit of randomization and justify it against the alternatives, define the overall evaluation criterion OEC the decision rests on, name the guardrail metrics that can block a launch even when the OEC wins, pick a minimum detectable effect and derive a duration from roughly 400k daily searches by 120k daily users, and say how you would stop the model from being retrained on the data it produced during the experiment.
Randomize by user, not by search, so a person consistently sees one ranker. The OEC is a single decision metric combining relevance and revenue; guardrails cover latency, seller coverage and complaints. Size on user-level variance and freeze the training data.
Common mistakes
- ✗Randomizing per request so one user sees both rankers
- ✗Treating the OEC as sufficient with no guardrail metrics
- ✗Letting the experiment logs feed the model under test
Follow-up questions
- →Which guardrail would trip first if the new ranker doubled the tail latency at the 99th percentile?
- →How would you detect that the challenger degraded long-tail queries while lifting the average?
MiddleTheoryCommonWhy does checking the p-value every morning inflate the false-positive rate, and what fixes peeking?
Why does checking the p-value every morning inflate the false-positive rate, and what fixes peeking?
Each look is another chance to cross the threshold, so repeated tests on the same accumulating data push the real false-positive rate far above the nominal 5 percent. Fixes are a fixed horizon, alpha-spending boundaries, or a valid sequential test.
Common mistakes
- ✗Treating an early crossing of 0.05 as a real result arriving sooner
- ✗Applying a plain Bonferroni over days as if the daily looks were independent
- ✗Stopping the test at the first significant reading
Follow-up questions
- →What does an alpha-spending function such as the boundary
O'Brien-Flemingdo to the early thresholds? - →Why can a sequential test be read at any moment without inflating the error rate?
SeniorDesignCommonYou are testing a new recommender on a marketplace where supply is finite: buyers shown the same listings compete for them, and a seller can sell an item only once. A user-level 50/50 split reports a 6 percent lift in treatment and the team wants to ship. Explain why user-level randomization is not trustworthy here, what the treatment group is doing to the control group, and which designs — a switchback over time windows, cluster randomization by region or seller, or a geo split — you would use instead, naming the cost each one carries.
You are testing a new recommender on a marketplace where supply is finite: buyers shown the same listings compete for them, and a seller can sell an item only once. A user-level 50/50 split reports a 6 percent lift in treatment and the team wants to ship. Explain why user-level randomization is not trustworthy here, what the treatment group is doing to the control group, and which designs — a switchback over time windows, cluster randomization by region or seller, or a geo split — you would use instead, naming the cost each one carries.
The units interfere: treatment users take the supply control users would have bought, so part of the lift is cannibalized from control and the estimate is biased upward. Randomize whole markets instead — switchback, cluster or geo — accepting far fewer effective units.
Common mistakes
- ✗Assuming random assignment alone removes interference bias
- ✗Reading a cannibalized lift as incremental value
- ✗Ignoring the loss of effective sample size in cluster designs
Follow-up questions
- →How does the seller side of the marketplace change what a cluster boundary must contain?
- →What does a switchback design assume about how quickly effects wash out between windows?
SeniorDebuggingCommonThe offline ranking metric ROC-AUC rose 3 percent but the online A/B is flat — how do you find where they disagree?
The offline ranking metric ROC-AUC rose 3 percent but the online A/B is flat — how do you find where they disagree?
Enumerate mechanically: training-serving skew and nulled features, leakage or a stale holdout offline, a label the offline metric rewards but the OEC never sees, gains outside the exposed slots, and dilution. Then replay production logs on one population.
Open full question →Common mistakes
- ✗Assuming a better offline metric must eventually show up online
- ✗Comparing offline and online numbers on different populations
- ✗Ignoring dilution from sessions that never saw the ranked list
Follow-up questions
- →How does nulled-feature behaviour at serving time change the ranking a user actually sees?
- →Why can a 7-day attribution window make a 14-day experiment read flat?
JuniorTheoryOccasionalWhat are novelty and primacy effects in a model A/B test, and why do they bias week-one results in opposite directions?
What are novelty and primacy effects in a model A/B test, and why do they bias week-one results in opposite directions?
Novelty is engagement with a change just because it is new, inflating early treatment metrics; primacy is the cost of relearning a familiar flow, depressing them. Both decay, so week-one numbers drift toward the truth — run longer and read by cohort day.
Common mistakes
- ✗Assuming both effects bias the week-one reading in the same direction
- ✗Reading the week-one lift as the steady-state effect
- ✗Believing the effects persist instead of decaying as users adapt
Follow-up questions
- →How does restricting the analysis to users new to the product help isolate novelty?
- →What does a lift curve that decays to zero over three weeks tell you about shipping?
JuniorTheoryOccasionalDo you show a p-value or a confidence interval on the lift, and how do you phrase a null result correctly?
Do you show a p-value or a confidence interval on the lift, and how do you phrase a null result correctly?
Show the confidence interval: it carries the effect size and its uncertainty, while a p-value only says whether zero is excluded. A null result means the interval still covers zero — the test could not detect the effect it was powered for.
Common mistakes
- ✗Reporting a non-significant result as proof of no difference
- ✗Reading the interval as a range for individual users rather than for the effect
- ✗Dropping the effect size and reporting significance alone
Follow-up questions
- →How does a wide interval spanning zero differ in decision value from a tight one?
- →When is a statistically significant lift still too small to ship?
SeniorTheoryOccasionalAcross 20 metrics, what does the correction Bonferroni cost in power, and when is false-discovery-rate control right?
Across 20 metrics, what does the correction Bonferroni cost in power, and when is false-discovery-rate control right?
Bonferroni divides alpha by the number of tests, controlling the chance of any false positive but sharply shrinking each test's power. Benjamini-Hochberg instead controls the expected share of false positives among rejections — right for exploratory scans.
Common mistakes
- ✗Correcting exploratory segment scans and the primary metric with the same procedure
- ✗Believing that a multiplicity correction can increase power
- ✗Confusing family-wise error control with false-discovery-rate control
Follow-up questions
- →How does declaring one primary metric in advance reduce the correction burden?
- →Why does a significant result in one of twenty segments rarely replicate?
SeniorTheoryRareWhich variance does variance-reduction method CUPED remove, what does it need from the pre-period, and when does it buy nothing?
Which variance does variance-reduction method CUPED remove, what does it need from the pre-period, and when does it buy nothing?
CUPED regresses the metric on a pre-period covariate and subtracts the predicted part, removing variance that pre-dates the split, not the effect. That covariate must correlate with the metric and stay untouched by treatment; weak correlation buys nothing.
Common mistakes
- ✗Thinking CUPED changes the treatment effect estimate rather than its variance
- ✗Using an in-experiment covariate that the treatment can influence
- ✗Expecting gains for users with no pre-period history
Follow-up questions
- →Why must the covariate be measured strictly before assignment happens?
- →How does the covariate correlation with the metric translate into a sample-size saving?
