The offline ranking metric ROC-AUC rose 3 percent but the online A/B is flat — how do you find where they disagree?
The offline ranking metric improved on the holdout, but the online launch experiment for the same model came back flat.
Constraints:
- both numbers were produced by the same model release;
- the splitter passed an A/A check one week ago;
- you may not redesign the experiment, only diagnose the gap.
offline (holdout, 2 weeks old) ROC-AUC 0.812 -> 0.836 (+3.0%)
online A/B (14 days, 1.2M users) CTR 2.41% -> 2.43% (p = 0.38)
revenue/user flat (p = 0.71)
serving log check features nulled at scoring: 4 of 37
label pipeline conversion attributed up to 7 days later
exposure 18% of sessions never reached the ranked listList the causes that can produce this gap and the order in which you would check them.
Enumerate mechanically: training-serving skew and nulled features, leakage or a stale holdout offline, a label the offline metric rewards but the OEC never sees, gains outside the exposed slots, and dilution. Then replay production logs on one population.
- ✗Assuming a better offline metric must eventually show up online
- ✗Comparing offline and online numbers on different populations
- ✗Ignoring dilution from sessions that never saw the ranked list
- →How does nulled-feature behaviour at serving time change the ranking a user actually sees?
- →Why can a 7-day attribution window make a 14-day experiment read flat?
Walkthrough
An offline-online gap almost always decomposes into five classes of cause, and it is cheapest to check them from the most mechanical to the most substantive.
different model than the one that was evaluated offline.
production no longer sees.
ordering gains outside the exposed slots never reach the OEC.
effect on affected users is divided across the whole arm.
tail is truncated.
- Training-serving skew. 4 of 37 features nulled at scoring time means production ranks with a
- A corrupted offline measurement. Leakage or a two-week-old holdout inflates ROC-AUC on data
- The wrong metric. ROC-AUC scores pairwise order, while the decision rests on CTR and revenue —
- Dilution. 18% of sessions never reached the ranked list yet stayed in the denominator, so the
- Attribution window. Conversions land up to 7 days later while the test ran 14 days, so the
The main instrument is replaying production logs offline so both metrics are read on one population:
exposed = logs[logs.reached_ranked_list] # remove dilution
served = exposed.features_served # exactly what serving saw, nulls included
offline_auc = roc_auc_score(exposed.label, model.predict(served))
online_ctr = exposed.groupby(exposed.variant).click.mean()
# if offline_auc on exposed drops to the production level, skew is the cause, not the experiment
