Linear models and optimization
A linear model predicts a weighted sum of features — a hyperplane in space. It is simple, yet it carries all the basic ML intuitions: what a loss function is, how to minimize it, what overfitting is, and how regularization restrains it. Get it here and half of the questions about complex models become obvious.
Two main traps in the topic. First, thinking that in production linear regression is solved by the closed-form formula: in fact gradient descent almost always does the work, because inverting a matrix costs O(n³) and is unstable. Second, treating L1 and L2 as "the same thing at a different scale": L1 zeroes weights and produces sparsity, L2 only shrinks them. The full map is in the layers below.
Topic map
- Linear regression — the hyperplane, squared loss, and how the task is solved in practice.
- Closed-form OLS — the solution
w = (XᵀX)⁻¹Xᵀy, its cost, and why it is avoided. - Regularization — a weight-norm penalty against overfitting and why the bias is left out of it.
- L1 and L2 — why L1 gives sparsity and L2 does not, and what Elastic Net is.
- Gradient descent — full-batch GD versus stochastic SGD and how to choose between them.
- GD variants — momentum, AdaGrad, RMSProp, Adam, and how they differ.
- SGD with momentum — accumulating gradients against the zig-zag on ill-conditioned level curves.
Common traps
| Mistake | Consequence |
|---|---|
| Thinking production runs the closed-form OLS formula | Ignores the O(n³) cost and instability when XᵀX is singular |
Calling matrix inversion O(n²) | Confuses entry count with operation count — it is O(n³) |
| Treating L1 and L2 as differing only in scale | Misses that only L1 produces sparsity |
| Penalizing the intercept (bias) | The hyperplane cannot be shifted to the data, quality drops |
| Thinking the penalty improves the training fit | On the contrary — the penalty restrains the fit for generalization |
| Confusing optimizers with regularization | Adam/RMSProp change the step, they do not penalize weights |
| Describing momentum as a fixed direction | It is a decaying running average of gradients, not one vector |
Interview relevance
Linear models are a favorite in interviews precisely because there is nowhere to hide: the OLS formula, the geometry of L1/L2, and the mechanics of gradient descent are either understood or not. A candidate who derives L1's sparsity from the constant penalty gradient, rather than a memorized diamond picture, immediately looks stronger.
Typical checks:
- How linear regression is solved in practice and why not by the closed-form formula.
- The OLS formula, the
O(n³)matrix inversion cost, and the role ofλI. - Why regularization is needed and why the intercept is excluded from the penalty.
- Where L1's sparsity comes from (diamond geometry + constant gradient) and why L2 has none.
- The optimizer family and what exactly momentum adds over SGD.
Common wrong answer: "production solves it via (XᵀX)⁻¹, and L1 and L2 are the same thing". In fact the formula is avoided for its cost and instability, and only L1 gives sparsity — L2 merely shrinks weights without driving them to zero.