Support Vector Machines
Maximum-margin classifiers and regressors: support vectors, soft margin C, the kernel trick, RBF gamma and the SVR epsilon-tube.
5 questions
JuniorTheoryCommonWhat does this program print?
What does this program print?
3 6
Open full question →Common mistakes
- ✗Expecting
6 3: a largeCmakes violations expensive and narrows the band, so few points touch it; a smallCwidens the band until every point is inside it. - ✗Expecting
3 3: support vectors are not only the points on the edge of the band; every point inside it or past it is one too, and a smallClets in more.
Follow-up questions
- →How does the width of the band change between
hardandsoft? - →Which value of
Cwould you try first on noisy data, and why?
JuniorTheoryCommonWhat does this program print?
What does this program print?
2 1
Open full question →Common mistakes
- ✗Expecting
1 1: the point[2, 1]lies on the line itself, itsmarginsvalue is0, which is below1, so it violates the margin too. - ✗Expecting
4 1: the points[1, 1]and[1, 3]give exactly1, which puts them on the edge of the band, andmargins < 1does not count them.
Follow-up questions
- →Which of these six points are support vectors of this line?
- →How wide is the band of this line, and how would doubling
wandbchange it?
JuniorTheoryCommonWhat does the kernel trick let a support vector machine (SVM) do, and what does it avoid computing?
What does the kernel trick let a support vector machine (SVM) do, and what does it avoid computing?
A kernel returns the inner product two points would have after a mapping into a higher-dimensional space, without building those features. The SVM needs only these pairwise products, so RBF or polynomial kernels give curved boundaries cheaply.
Common mistakes
- ✗Picturing the kernel as building and storing the mapped features — it returns only their inner product, which is why the RBF kernel's infinite-dimensional space is usable.
- ✗Assuming a kernel SVM still draws a straight line in the input space — it is linear only in the implicit feature space, so the input-space boundary can curve.
Follow-up questions
- →Why does the cost of a kernel SVM grow with the number of training samples?
- →Which kernel would you try first on data you know little about?
JuniorTheoryCommonHow does gamma of an RBF (radial basis function) kernel shape an SVM boundary, and does scaling matter?
How does gamma of an RBF (radial basis function) kernel shape an SVM boundary, and does scaling matter?
Large RBF gamma shrinks each support vector's influence to a small neighborhood, giving a wiggly boundary that overfits; small gamma spreads it wide and smooths the boundary. RBF distances need scaled features, or one feature dominates.
Common mistakes
- ✗Reading
gammaas the margin width — it sets how fast each point's influence decays with distance, and a large value makes the boundary hug individual points. - ✗Skipping scaling for an RBF SVM — its kernel uses Euclidean distance, so a feature measured in thousands drowns out one measured in units.
Follow-up questions
- →How would you tune
gammatogether with C? - →What does an RBF SVM predict far from every support vector when
gammais large?
JuniorTheoryCommonIn support vector regression (SVR), what does the ε-tube do to the fit and to the support vectors?
In support vector regression (SVR), what does the ε-tube do to the fit and to the support vectors?
SVR charges no loss for points whose error stays inside the ε-tube. Only points on or outside the tube become support vectors and shape the fit, so widening ε leaves fewer support vectors and a flatter, simpler function.
Common mistakes
- ✗Treating ε as a learning rate — it is the half-width of the zero-loss tube, and it decides which points become support vectors and how flat the fit is.
- ✗Expecting every training point to shape an SVR fit — points strictly inside the tube carry zero loss and drop out of the solution.
Follow-up questions
- →How would you choose ε relative to the noise level in the target?
- →What happens to the number of support vectors as ε approaches zero?
