SeniorDebuggingCommonNot answered yet
The offline ranking metric ROC-AUC rose 3 percent but the online A/B is flat — how do you find where they disagree?
The offline ranking metric improved on the holdout, but the online launch experiment for the same model came back flat.
Constraints:
- both numbers were produced by the same model release;
- the splitter passed an A/A check one week ago;
- you may not redesign the experiment, only diagnose the gap.
offline (holdout, 2 weeks old) ROC-AUC 0.812 -> 0.836 (+3.0%)
online A/B (14 days, 1.2M users) CTR 2.41% -> 2.43% (p = 0.38)
revenue/user flat (p = 0.71)
serving log check features nulled at scoring: 4 of 37
label pipeline conversion attributed up to 7 days later
exposure 18% of sessions never reached the ranked listList the causes that can produce this gap and the order in which you would check them.
Enumerate mechanically: training-serving skew and nulled features, leakage or a stale holdout offline, a label the offline metric rewards but the OEC never sees, gains outside the exposed slots, and dilution. Then replay production logs on one population.
- ✗Assuming a better offline metric must eventually show up online
- ✗Comparing offline and online numbers on different populations
- ✗Ignoring dilution from sessions that never saw the ranked list
- →How does nulled-feature behaviour at serving time change the ranking a user actually sees?
- →Why can a 7-day attribution window make a 14-day experiment read flat?
Walkthrough
An offline-online gap almost always decomposes into five classes of cause, and it is cheapest to check them from the most mechanical to the most substantive.
- Training-serving skew. 4 of 37 features nulled at scoring time means production ranks with a different model than the one that was evaluated offline.
- A corrupted offline measurement. Leakage or a two-week-old holdout inflates ROC-AUC on data production no longer sees.
- The wrong metric. ROC-AUC scores pairwise order, while the decision rests on CTR and revenue — ordering gains outside the exposed slots never reach the OEC.
- Dilution. 18% of sessions never reached the ranked list yet stayed in the denominator, so the effect on affected users is divided across the whole arm.
- Attribution window. Conversions land up to 7 days later while the test ran 14 days, so the tail is truncated.
The main instrument is replaying production logs offline so both metrics are read on one population:
exposed = logs[logs.reached_ranked_list] # remove dilution
served = exposed.features_served # exactly what serving saw, nulls included
offline_auc = roc_auc_score(exposed.label, model.predict(served))
online_ctr = exposed.groupby(exposed.variant).click.mean()
# if offline_auc on exposed drops to the production level, skew is the cause, not the experiment
