MiddleDebuggingCommonNot answered yet

CV says 0.92 and production says 0.71 — which leakage channels explain the gap?

A fraud model scores ROC-AUC 0.92 in cross-validation and 0.71 in production.

Constraints:

df holds one row per transaction, ordered by txn_at; a user has many transactions.
chargeback_at is written only when a transaction is later disputed.
Roughly 4% of rows are exact retries of another row.

df["days_to_chargeback"] = (df["chargeback_at"] - df["txn_at"]).dt.days
X = df.drop(columns=["is_fraud"])
y = df["is_fraud"]

scores = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))

Name every leakage channel present and give the fix for each.

Four channels: target leakage from a field filled only after the outcome, contamination from duplicate rows spanning folds, group leakage from one user on both sides, and temporal leakage from shuffling time-ordered data. Fixing one alone drops the score.

✗Blaming drift for a gap that a post-outcome feature fully explains
✗Assuming a shuffled split is safe on data that is ordered in time
✗Trusting an averaged fold score while duplicates span the folds

→How would you rank the four channels by the damage each one usually does?
→What audit would catch a post-outcome feature before any model is trained?

Four channels explain the gap.

Target leakage. days_to_chargeback is derived from chargeback_at, which is written only after a dispute. Honest transactions have a missing value there, so the model reads the label directly. Drop the field or recompute it as of prediction time.
Duplicate contamination. The 4% exact retries land in different folds, so the model recognises a row it has already seen. Deduplicate before splitting.
Group leakage. A user has many transactions and KFold puts them on both sides. Fold by user_id group.
Temporal leakage. shuffle=True on time-ordered data trains on the future. Split by time instead.