Classical ML & Boosting

Decision trees and why single trees overfit, gradient boosting with XGBoost/LightGBM/CatBoost, and unsupervised learning — clustering and dimensionality reduction.

IT Abyss

Classical ML & Boosting

Before transformers, and still today on tabular data, the workhorse is a decision tree and an ensemble of them. This topic builds the mental model an interviewer probes: how a single tree splits and why it overfits, how gradient boosting stacks weak trees into the strongest tabular model we have, and how unsupervised methods find structure when there are no labels at all. Deep learning wins on text and images; on a spreadsheet of features, gradient-boosted trees usually still win.

The three layers below move from one tree, to a library of boosted trees, to learning without a target. Along the way we name the traps that separate a confident answer from a recited one: a single tree extrapolates to a flat line, boosting fits the negative gradient — not the raw data — one tree at a time, and a t-SNE picture is for the eye, never for a downstream model.

Topic map

Decision trees — how a node splits on impurity (Gini/entropy), why an unbounded tree memorises, pruning and depth, and why impurity feature-importance is biased toward high-cardinality columns.
Boosting libraries — gradient boosting as sequential fitting of the negative gradient, XGBoost vs LightGBM vs CatBoost (level-wise vs leaf-wise, native categoricals), and the hyperparameters that control overfitting.
Unsupervised learning — k-means and its spherical assumption, DBSCAN and hierarchical clustering, PCA's max-variance directions, and why t-SNE/UMAP are visualisation only.

Interview relevance

Classical ML is asked as a test of your model, not your API knowledge. A candidate who says "boosting fits each tree to the residual of the last one, sequentially" is instantly ahead of one who says "it averages a lot of trees". The recurring checks: what impurity a split minimises, which knobs stop a tree memorising, why leaf-wise growth overfits small data, and why PCA and t-SNE are not interchangeable.

advanced

Classical ML & Boosting

Decision trees and why single trees overfit, gradient boosting with XGBoost/LightGBM/CatBoost, and unsupervised learning — clustering and dimensionality reduction.

ml-classical

Practice: 11 questions →

Sections

Contents

Detailed explanation

Classical ML & Boosting

Topic map

Decision trees — how a node splits on impurity (Gini/entropy), why an unbounded tree memorises, pruning and depth, and why impurity feature-importance is biased toward high-cardinality columns.
Boosting libraries — gradient boosting as sequential fitting of the negative gradient, XGBoost vs LightGBM vs CatBoost (level-wise vs leaf-wise, native categoricals), and the hyperparameters that control overfitting.
Unsupervised learning — k-means and its spherical assumption, DBSCAN and hierarchical clustering, PCA's max-variance directions, and why t-SNE/UMAP are visualisation only.

Interview relevance

Why it matters

On tabular data the strongest model is still an ensemble of trees, not a neural network, so this is where most production ML and most interviews actually live. Confuse boosting with bagging, claim a tree extrapolates a trend, or feed t-SNE coordinates into a classifier, and you signal that you recite libraries without the model underneath. This topic gives you that model.

Classical ML & Boosting

Decision trees and why single trees overfit, gradient boosting with XGBoost/LightGBM/CatBoost, and unsupervised learning — clustering and dimensionality reduction.

IT Abyss

Classical ML & Boosting

Topic map

Decision trees — how a node splits on impurity (Gini/entropy), why an unbounded tree memorises, pruning and depth, and why impurity feature-importance is biased toward high-cardinality columns.
Boosting libraries — gradient boosting as sequential fitting of the negative gradient, XGBoost vs LightGBM vs CatBoost (level-wise vs leaf-wise, native categoricals), and the hyperparameters that control overfitting.
Unsupervised learning — k-means and its spherical assumption, DBSCAN and hierarchical clustering, PCA's max-variance directions, and why t-SNE/UMAP are visualisation only.

Interview relevance

Classical ML & Boosting

Topic map

Decision trees — how a node splits on impurity (Gini/entropy), why an unbounded tree memorises, pruning and depth, and why impurity feature-importance is biased toward high-cardinality columns.

Boosting libraries — gradient boosting as sequential fitting of the negative gradient, XGBoost vs LightGBM vs CatBoost (level-wise vs leaf-wise, native categoricals), and the hyperparameters that control overfitting.

Unsupervised learning — k-means and its spherical assumption, DBSCAN and hierarchical clustering, PCA's max-variance directions, and why t-SNE/UMAP are visualisation only.

Interview relevance