A CatBoost model scores 0.94 ROC-AUC in cross-validation and 0.78 in production — find the cause.
A fraud model scores 0.94 ROC-AUC in cross-validation and 0.78 on the first week in production, on exactly the same feature list.
Constraints:
orders.csvis an 18-month event log with anevent_timecolumn.merchant_idhas about 40 000 distinct values.- Production scores each order once, at the moment it arrives.
import pandas as pd
from catboost import CatBoostClassifier
from sklearn.model_selection import cross_val_score
df = pd.read_csv("orders.csv")
y = df.pop("is_fraud")
df["merchant_te"] = y.groupby(df["merchant_id"]).transform("mean")
df = df.drop(columns=["merchant_id", "event_time"])
model = CatBoostClassifier(iterations=2000, verbose=0)
print(cross_val_score(model, df, y, cv=5, scoring="roc_auc").mean()) # 0.94Find the defect and fix the evaluation.
The merchant mean is computed over all labels before splitting, so each fold's encoding contains its own targets — leakage, not CV overfitting. Pass merchant_id in cat_features so ordered statistics encode it, and validate by time.
- ✗Blaming CV overfitting when the encoder has seen the labels of every row
- ✗Assuming CatBoost's ordered statistics protect hand-made numeric encodings
- ✗Validating time-ordered events with random K-fold splits
- →How would you compute a leak-free target encoding by hand?
- →Which validation scheme matches an 18-month event log?
merchant_te is computed over the whole y column before any split, so each row's own label already sits inside its own feature. Every cross-validation fold is scored on data whose targets leaked into it — hence the 0.94. Production cannot build that feature at all, because the label is not yet known. On top of that, a random cv=5 shuffles 18 months of events and lets the model train on the future.
The fix is to hand the raw category to CatBoost (its ordered target statistics encode a row only from rows earlier in the permutation) and to validate with a time-based split.
import pandas as pd
from catboost import CatBoostClassifier
from sklearn.metrics import roc_auc_score
df = pd.read_csv("orders.csv").sort_values("event_time")
y = df.pop("is_fraud")
cut = int(len(df) * 0.8)
X = df.drop(columns=["event_time"])
X_tr, X_va = X.iloc[:cut], X.iloc[cut:]
y_tr, y_va = y.iloc[:cut], y.iloc[cut:]
model = CatBoostClassifier(iterations=2000, early_stopping_rounds=100, verbose=0)
model.fit(X_tr, y_tr, cat_features=["merchant_id"], eval_set=(X_va, y_va))
print(roc_auc_score(y_va, model.predict_proba(X_va)[:, 1]))The resulting AUC lands near the production number, because the evaluation finally reproduces production conditions.
