A scaler, imputer and selector are fit before the split — what exactly leaks, and how?
The cross-validation score of this classifier is far above its held-out score.
Constraints:
Xis a wide table with missing values;yis the binary target.- The model itself and the fold count must stay as they are.
- Fix only how the preprocessing steps are fit.
X = StandardScaler().fit_transform(X)
X = SimpleImputer(strategy="mean").fit_transform(X)
X = SelectKBest(f_classif, k=20).fit_transform(X, y)
scores = cross_val_score(model, X, y, cv=5)Say what leaks at each step and rewrite the code so nothing is fit outside a training fold.
Fitting them on the whole table pushes held-out rows into the training statistics: the scaler carries held-out means and variances, the imputer fills from them, and selection reads held-out labels, which leaks worst. A pipeline refits each step per training fold.
- ✗Calling unsupervised transforms leak-free because they never read the target
- ✗Scaling once up front for convenience and splitting the array afterwards
- ✗Selecting features on the full table and then cross-validating the survivors
- →Why does target encoding need an inner split even inside a correct pipeline?
- →How do you apply the same discipline to a resampling step such as
SMOTE?
Every step leaks, each in its own way.
StandardScalerfit on the whole table computes the mean and variance over held-out rows too, so the training fold learns the scale of data it should not have seen.SimpleImputerfills gaps with a mean computed partly from held-out rows.SelectKBestis the worst case: it readsyover all rows, so the 20 surviving features are already tuned to the held-out labels. The score after such a selection is the most inflated of the three.
The fix is to assemble the steps into a Pipeline and hand that to the cross-validator. Then each step's fit runs on the training fold only, and the held-out fold sees transform alone.
pipe = Pipeline([
("impute", SimpleImputer(strategy="mean")),
("scale", StandardScaler()),
("select", SelectKBest(f_classif, k=20)),
("model", model),
])
scores = cross_val_score(pipe, X, y, cv=5)Imputation comes before scaling, otherwise the mean is computed over a table that still has gaps.
