Offline NDCG rose but online click-through rate fell — fix this evaluation script.
An offline evaluation reports a large NDCG gain for a new ranker, but the online A/B test shows a click-through rate drop. The log holds one row per impression served by the current production ranker, with a session_id, a served_at timestamp and a binary clicked label.
Constraints — the log is the only data available, the ranker itself is known to be sound, and the metric implementation below is numerically correct.
logs = pd.read_parquet("impressions.parquet") # only what the live ranker showed
train, test = train_test_split(logs, test_size=0.2, random_state=0)
model = fit_ranker(train)
def session_ndcg(df, k=10):
scored = df.assign(score=model.predict(df)).sort_values("score", ascending=False)
top = scored.head(k)
gains = top["clicked"] / np.log2(np.arange(2, len(top) + 2))
return gains.sum() / ideal_dcg(df, k)
print(logs.groupby("session_id").apply(session_ndcg).mean())Find the three flaws that let offline NDCG rise while online click-through rate falls, and fix them.
It scores the full log including trained-on rows, splits randomly so future sessions leak back, and trusts logged clicks though only what the old ranker exposed was seen. Split by time, score held-out sessions, and reweight by inverse propensity.
- ✗Random-splitting interaction logs instead of splitting by time
- ✗Scoring the metric on rows the model was trained on
- ✗Treating logged clicks as unbiased relevance labels
- →How would you build a counterfactual estimator that better predicts online lift?
- →Which of the three flaws would you expect to inflate offline NDCG the most?
The script breaks in three places at once, and all three inflate the offline number.
session land on both sides.
click at the top reflects exposure as much as relevance.
- The metric runs over
logs, nottest— the model scores rows it was trained on. train_test_splitsplits randomly — future sessions leak into training, and rows of one- Clicks are taken as unbiased labels — the log contains only what the old ranker showed, and a
cutoff = logs["served_at"].quantile(0.8)
train = logs[logs["served_at"] < cutoff]
test = logs[logs["served_at"] >= cutoff]
model = fit_ranker(train)
def session_ndcg(df, k=10):
scored = df.assign(score=model.predict(df)).sort_values("score", ascending=False)
top = scored.head(k)
weights = 1.0 / top["propensity"] # inverse examination propensity
gains = top["clicked"] * weights / np.log2(np.arange(2, len(top) + 2))
return gains.sum() / ideal_dcg(df, k)
print(test.groupby("session_id").apply(session_ndcg).mean())Propensities are estimated from randomized serving slots. Even then the offline number stays an estimate — the A/B test makes the final call.
