Offline NDCG rose but online click-through rate fell — fix this evaluation script.
An offline evaluation reports a large NDCG gain for a new ranker, but the online A/B test shows a click-through rate drop. The log holds one row per impression served by the current production ranker, with a session_id, a served_at timestamp and a binary clicked label.
Constraints — the log is the only data available, the ranker itself is known to be sound, and the metric implementation below is numerically correct.
logs = pd.read_parquet("impressions.parquet") # only what the live ranker showed
train, test = train_test_split(logs, test_size=0.2, random_state=0)
model = fit_ranker(train)
def session_ndcg(df, k=10):
scored = df.assign(score=model.predict(df)).sort_values("score", ascending=False)
top = scored.head(k)
gains = top["clicked"] / np.log2(np.arange(2, len(top) + 2))
return gains.sum() / ideal_dcg(df, k)
print(logs.groupby("session_id").apply(session_ndcg).mean())Find the three flaws that let offline NDCG rise while online click-through rate falls, and fix them.
It scores the full log including trained-on rows, splits randomly so future sessions leak back, and trusts logged clicks though only what the old ranker exposed was seen. Split by time, score held-out sessions, and reweight by inverse propensity.
- ✗Random-splitting interaction logs instead of splitting by time
- ✗Scoring the metric on rows the model was trained on
- ✗Treating logged clicks as unbiased relevance labels
- →How would you build a counterfactual estimator that better predicts online lift?
- →Which of the three flaws would you expect to inflate offline NDCG the most?
The script breaks in three places at once, and all three inflate the offline number.
- The metric runs over
logs, nottest— the model scores rows it was trained on. train_test_splitsplits randomly — future sessions leak into training, and rows of one session land on both sides.- Clicks are taken as unbiased labels — the log contains only what the old ranker showed, and a click at the top reflects exposure as much as relevance.
cutoff = logs["served_at"].quantile(0.8)
train = logs[logs["served_at"] < cutoff]
test = logs[logs["served_at"] >= cutoff]
model = fit_ranker(train)
def session_ndcg(df, k=10):
scored = df.assign(score=model.predict(df)).sort_values("score", ascending=False)
top = scored.head(k)
weights = 1.0 / top["propensity"] # inverse examination propensity
gains = top["clicked"] * weights / np.log2(np.arange(2, len(top) + 2))
return gains.sum() / ideal_dcg(df, k)
print(test.groupby("session_id").apply(session_ndcg).mean())Propensities are estimated from randomized serving slots. Even then the offline number stays an estimate — the A/B test makes the final call.
