CV score jumps after adding SMOTE, but production quality does not move
Fraud detection, roughly 1 positive per 1000 sessions. After adding the oversampling technique SMOTE, cross-validated ROC-AUC rose from 0.81 to 0.95, but production alert quality did not change at all.
Constraint: the model, the features and the CV scheme are unchanged; only the resampling step was added.
X_res, y_res = SMOTE(random_state=0).fit_resample(X, y)
scores = cross_val_score(model, X_res, y_res,
cv=StratifiedKFold(5), scoring="roc_auc")
print(scores.mean()) # 0.95, was 0.81Diagnose the cause.
You resampled before splitting: synthetic points built from training rows land in the validation fold, so CV grades near-copies of what the model saw and inflates. Resample inside each fold, on the training part only — put SMOTE in a pipeline CV refits per fold.
- ✗Calling
fit_resampleon the full dataset before the train/validation split - ✗Assuming synthetic rows cannot leak because they are new objects
- ✗Resampling the validation or test fold to match the training balance
- →Which class keeps a resampler inside the fold in the library
imbalanced-learn? - →Would class weights instead of SMOTE have avoided this leak entirely?
The leak comes from the order of operations. fit_resample runs on the whole dataset before the split: SMOTE builds a synthetic point on the segment between two real minority objects, and both parents and their child are scattered across different folds. The validation fold receives near-copies of training rows, ROC-AUC climbs, and production sees no such pair — so nothing improves there.
The fix is to move resampling inside the fold: the imbalanced-learn Pipeline applies the resampler to the training part of each split only and never touches the validation part.
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
pipe = Pipeline([("smote", SMOTE(random_state=0)), ("model", model)])
scores = cross_val_score(pipe, X, y, cv=StratifiedKFold(5), scoring="roc_auc")
print(scores.mean()) # back to about 0.81 — an honest estimate
