Ensembles
An ensemble composes many weak models into one strong model. The point is that the individual models' errors are partly independent, and a good combination lets them cancel out. That is precisely why tree ensembles — Random Forest and gradient boosting — have stayed the top choice on tabular data for years, beating single models and often neural networks.
The two big branches are built the opposite way, and confusing them is the main trap of the topic. Bagging trains models in parallel and independently, averages them, and reduces variance. Boosting trains models sequentially, each correcting the errors of the previous ones, and reduces bias. Everything else follows from this difference: why Random Forest does not overfit with more trees while boosting does, why bagging linear models is pointless, and where L1/L2 regularization hides inside GBDT. The full map is in the layers below.
Topic map
- What an ensemble is — the bias-variance decomposition and why combining models lowers error.
- Bagging — bootstrap aggregation, variance reduction by averaging, and why bagging linear models gives nothing.
- Random Forest — bagging over trees plus a random feature subspace chosen at every node.
- Boosting — a sequential additive composition where each model corrects the previous ones and reduces bias.
- Gradient boosting — fitting the negative gradient of the loss as descent in function space, the learning rate, and its tendency to overfit.
- GBDT implementations — CatBoost, LightGBM, and XGBoost and how they build their trees differently.
- GBDT hyperparameters — depth, learning rate, leaf size, and L1/L2 on leaf values.
- Hyperparameter tuning — grid/random/Bayesian search, validation, and the curse of dimensionality.
- Stacking — a meta-model over base models' out-of-fold predictions and how it differs from blending.
Common traps
| Mistake | Consequence |
|---|---|
| Confusing bagging (parallel) with boosting (sequential) | Wrong conclusions about overfitting, bias, and variance across the topic |
| Claiming bagging reduces bias | Bagging reduces variance; bias stays the same |
| Expecting an N× variance drop always | It holds only for uncorrelated models, which is not the case in practice |
| Thinking Random Forest overfits with more trees | Boosting overfits; RF barely changes with extra trees |
| Fitting boosting trees to the raw labels | Boosting's target is the negative gradient at the current prediction |
| Saying trees have no weights for L1/L2 | The leaf values are penalized — they play the role of the weights |
| Training a stacking meta-model on in-sample predictions | Target leakage; out-of-fold predictions are required |
Interview relevance
Ensembles are one of the most common ML interview blocks because they are good for probing depth: memorizing definitions is easy, explaining why boosting overfits while RF does not is not. A strong candidate runs the whole argument off one difference: bagging kills variance by averaging independent models, boosting kills bias by sequentially correcting errors.
Typical checks:
- The difference between bagging and boosting and what each does to bias and variance.
- How Random Forest is built — and that the feature subset is drawn at each node, not once per tree.
- That the training target in gradient boosting is the negative gradient, not residuals in the narrow sense.
- GBDT specifics: implementations, key hyperparameters, and where exactly L1/L2 lives.
- Stacking versus blending and why out-of-fold predictions are mandatory.
Common wrong answer: "bagging reduces both bias and variance, and Random Forest is a kind of boosting". In fact bagging touches only variance, and Random Forest is parallel, not sequential, which is exactly why it does not overfit as the tree count grows.