Decision Trees
Single-tree mechanics: split criteria, impurity, pruning, extrapolation limits, feature-importance bias, and native missing-value handling.
9 questions
JuniorTheoryVery commonHow do split criteria Gini, entropy and mean squared error differ in practice?
How do split criteria Gini, entropy and mean squared error differ in practice?
Gini and entropy both score class mixture and rank splits almost identically; entropy adds a log and costs a bit more. Regression trees minimise MSE, so a leaf predicts the mean, or MAE for the median. Depth and pruning matter far more.
Common mistakes
- ✗Expecting a large accuracy gain from switching Gini to entropy
- ✗Using MSE on a heavy-tailed target and expecting median-like robustness
- ✗Thinking a regression leaf fits a line rather than a constant
Follow-up questions
- →Why does a leaf predicting the mean follow directly from minimising MSE?
- →When would you deliberately pick MAE over MSE for a regression tree?
JuniorTheoryCommonHow can a tree handle missing values without imputing them first?
How can a tree handle missing values without imputing them first?
Two mechanisms. A learned default direction sends every missing value of the split feature down one branch — training tries both sides and keeps the better gain, as LightGBM and XGBoost do. Surrogate splits, used by CART, route the row by a backup feature.
Common mistakes
- ✗Assuming every library imputes silently, so missing handling never needs checking
- ✗Thinking the default direction is fixed rather than learned from the gain
- ✗Confusing a surrogate split with a duplicate of the primary split
Follow-up questions
- →What happens at prediction time if a feature was never missing during training?
- →When is explicit imputation still preferable to the native handling?
JuniorTheoryCommonHow do pre-pruning and post-pruning differ, and what is cost-complexity pruning?
How do pre-pruning and post-pruning differ, and what is cost-complexity pruning?
Pre-pruning bounds growth up front via max_depth, min_samples_leaf or min_impurity_decrease — cheap, but it can stop before a useful split appears. Post-pruning grows the tree fully, then drops subtrees whose error gain does not pay for their size.
Common mistakes
- ✗Treating max_depth as the only pruning knob and ignoring leaf-size limits
- ✗Selecting the cost-complexity alpha on training data instead of validation
- ✗Assuming post-pruning always beats a plain depth cap
Follow-up questions
- →Why can pre-pruning miss a split that only becomes useful two levels deeper?
- →How would you pick the cost-complexity alpha with cross-validation?
JuniorTheoryCommonDo decision trees need feature scaling or one-hot encoding, and why?
Do decision trees need feature scaling or one-hot encoding, and why?
No scaling is needed — a split compares one feature to a threshold, so monotone rescaling keeps the ordering and every candidate split identical. Categories need a numeric code, but ordinal codes work; one-hot is optional and with many levels dilutes each category.
Common mistakes
- ✗Standardising features before a tree and expecting a different model
- ✗One-hot encoding a high-cardinality column and wondering why splits weaken
- ✗Believing a tree cannot use an integer-coded category at all
Follow-up questions
- →Why does a non-monotone transform change a tree while a monotone one does not?
- →When would one-hot encoding still help a tree despite the dilution?
JuniorTheoryCommonHow does a decision tree choose a split, and what does Gini impurity measure?
How does a decision tree choose a split, and what does Gini impurity measure?
At each node the tree tries every feature and every candidate threshold, scores each split by the impurity of its two children weighted by size, and keeps the largest impurity drop. Gini is the chance of mislabelling a random object by a random class draw.
Common mistakes
- ✗Thinking splits minimise error directly rather than node impurity
- ✗Forgetting the children's impurity is weighted by their sample counts
- ✗Reading a high Gini as a pure node — 0 is pure, higher is mixed
Follow-up questions
- →Why is the split search greedy and local rather than globally optimal?
- →How does the threshold scan change for a continuous feature with many unique values?
MiddleDebuggingCommonImpurity importance ranks a near-unique ID column first — what is wrong, and what replaces it?
Impurity importance ranks a near-unique ID column first — what is wrong, and what replaces it?
Impurity importance sums the impurity a feature removed, and a high-cardinality column offers far more candidate thresholds, so it wins splits by chance without generalising — and it is read off the training fit. Use permutation importance on a held-out split instead.
Open full question →Common mistakes
- ✗Trusting
feature_importances_as a causal or generalisable ranking - ✗Assuming a high impurity importance implies a high validation contribution
- ✗Running permutation importance on the training split, which reproduces the bias
Follow-up questions
- →Why do correlated features still confuse permutation importance, and what helps?
- →How does dropping a feature and refitting differ from permuting it?
MiddleTheoryCommonThe root split is nearly perfect on one feature — leak or genuinely strong predictor?
The root split is nearly perfect on one feature — leak or genuinely strong predictor?
Check the feature against the data dictionary and the timeline — if its value is known only after the target event, or is derived from it, that is leakage. A genuine predictor keeps its edge on a later time slice; a leak collapses in production.
Common mistakes
- ✗Judging a feature by correlation strength instead of availability at prediction time
- ✗Trusting a random validation split that shares the leaked column with training
- ✗Assuming any column present in the training table exists at scoring time
Follow-up questions
- →How does a time-based validation split expose a leak that random k-fold hides?
- →Which columns in an events table are most often derived from the target?
MiddleDebuggingCommonA tree scores 100% on train and 60% on test — which knobs help, and how do you verify?
A tree scores 100% on train and 60% on test — which knobs help, and how do you verify?
The tree memorised the data — unbounded depth lets leaves hold single rows. Cap max_depth, raise min_samples_leaf, or post-prune by sweeping ccp_alpha. Verify with a cross-validated score, not the leaf count — pruning helped only if held-out score rises and the gap narrows.
Open full question →Common mistakes
- ✗Reading a smaller tree as proof that pruning improved generalisation
- ✗Tuning the depth cap on the test split and then reporting that same score
- ✗Relying on max_depth alone and ignoring leaf-size limits
Follow-up questions
- →How would you sweep ccp_alpha with cross-validation rather than a single split?
- →Which knob would you reach for first when the classes are strongly imbalanced?
MiddleTheoryOccasionalWhy can a decision tree never extrapolate beyond its training range?
Why can a decision tree never extrapolate beyond its training range?
A tree predicts one constant per leaf, so its output is piecewise-constant. Anything beyond the largest training value lands in the same edge leaf and gets that constant — flat, never a trend. For a trending feature, detrend the target and let the tree fit the residual.
Common mistakes
- ✗Feeding a raw time or trend feature into a tree and expecting future growth
- ✗Thinking more depth or more trees can restore extrapolation
- ✗Confusing the flat edge-leaf prediction with a returned global mean
Follow-up questions
- →How does detrending or differencing the target restore usable behaviour?
- →Does a random forest or gradient boosting change this limit, and why not?
