Impurity importance ranks a near-unique ID column first — what is wrong, and what replaces it?
A fitted tree puts a near-unique user_id column at the top of feature_importances_, yet dropping the column barely moves the validation score.
Constraints: do not collect new data and do not change the feature set; diagnose using only the fitted model and an existing held-out split.
importances = model.feature_importances_
ranking = sorted(zip(X_train.columns, importances), key=lambda p: -p[1])
print(ranking[:5]) # user_id dominates the ranking
# TODO: explain the bias and produce a trustworthy ranking insteadDiagnose the bias and replace the ranking.
Impurity importance sums the impurity a feature removed, and a high-cardinality column offers far more candidate thresholds, so it wins splits by chance without generalising — and it is read off the training fit. Use permutation importance on a held-out split instead.
- ✗Trusting
feature_importances_as a causal or generalisable ranking - ✗Assuming a high impurity importance implies a high validation contribution
- ✗Running permutation importance on the training split, which reproduces the bias
- →Why do correlated features still confuse permutation importance, and what helps?
- →How does dropping a feature and refitting differ from permuting it?
The bias comes out of the split search itself. A near-unique column offers almost as many candidate thresholds as there are rows, so one of them lowers impurity noticeably by chance. The tree credits that drop to the column, and feature_importances_ — computed on the training fit — floats user_id to the top with zero generalisation behind it.
A trustworthy ranking comes from permutation importance on a held-out split: shuffle the column and measure how far the score falls.
from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_valid, y_valid, n_repeats=20, random_state=0)
order = r.importances_mean.argsort()[::-1]
for i in order[:5]:
print(X_valid.columns[i], round(r.importances_mean[i], 4))
# user_id sinks toward zero: shuffling it does not hurt the scoreA feature whose shuffling costs nothing contributes nothing. Keep user_id out of the feature set entirely — as an identifier it is also a leakage risk.
