A tree scores 100% on train and 60% on test — which knobs help, and how do you verify?
A DecisionTreeClassifier fitted with default settings reaches a perfect training score and roughly 60% on the held-out split.
Constraints: keep the same features and the same data; change only the model's complexity controls, and judge every change on held-out data rather than on the fit.
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(tree.score(X_train, y_train)) # 1.00
print(tree.score(X_test, y_test)) # 0.60
print(tree.get_n_leaves()) # very many
# TODO: bound the tree and prove the fix on held-out dataFind the knobs and verify the fix correctly.
The tree memorised the data — unbounded depth lets leaves hold single rows. Cap max_depth, raise min_samples_leaf, or post-prune by sweeping ccp_alpha. Verify with a cross-validated score, not the leaf count — pruning helped only if held-out score rises and the gap narrows.
- ✗Reading a smaller tree as proof that pruning improved generalisation
- ✗Tuning the depth cap on the test split and then reporting that same score
- ✗Relying on max_depth alone and ignoring leaf-size limits
- →How would you sweep ccp_alpha with cross-validation rather than a single split?
- →Which knob would you reach for first when the classes are strongly imbalanced?
A 100% versus 60% gap is textbook memorisation: with unbounded depth the tree keeps splitting until leaves hold a single row, and such a leaf describes noise rather than a pattern.
There are two ways to bound complexity. Pre-pruning — max_depth, min_samples_leaf, min_impurity_decrease. Post-pruning — sweeping ccp_alpha along the cost-complexity path.
from sklearn.model_selection import cross_val_score
path = tree.cost_complexity_pruning_path(X_train, y_train)
for alpha in path.ccp_alphas[::5]:
m = DecisionTreeClassifier(random_state=0, ccp_alpha=alpha)
print(round(alpha, 5), round(cross_val_score(m, X_train, y_train, cv=5).mean(), 3))The verification criterion is the point. A smaller leaf count proves nothing — you can prune to a stump and get fewer leaves with worse quality. Pruning worked if the cross-validated score rose and the train-test gap narrowed. The test split is touched once, at the very end.
