All topics
k-Nearest Neighbors
Distance-based prediction: choosing k, distance metrics, why feature scaling is mandatory, and the cost of a lazy learner.
Sort:
4 questions
JuniorTheoryCommonHow does the choice of k in k-nearest neighbors (k-NN) affect overfitting, and how do you pick it?
JuniorTheoryCommon
How does the choice of k in k-nearest neighbors (k-NN) affect overfitting, and how do you pick it?
With k = 1 every training point is its own nearest neighbor, so the model memorizes the training set and overfits. A large k smooths predictions toward the majority class and underfits. Choose k by cross-validation.
Common mistakes
- ✗Assuming more neighbors always means more accuracy — past some k the vote drifts toward the majority class and the model underfits.
- ✗Picking k by training accuracy — that always favors k = 1, which memorizes the data; compare k values on a validation set instead.
Follow-up questions
- →Why is an odd k often preferred for binary classification?
- →How does distance weighting change the effect of a large k?
JuniorTheoryCommonWhy does k-nearest neighbors (k-NN) need feature scaling, and does a decision tree need it too?
JuniorTheoryCommon
Why does k-nearest neighbors (k-NN) need feature scaling, and does a decision tree need it too?
k-NN ranks neighbors by distance, so the feature with the largest range dominates — income in rubles swamps age in years. Scale features first. A decision tree splits on one feature's threshold at a time, so it does not need scaling.
Common mistakes
- ✗Assuming k-NN is scale-invariant like a tree — its distance sums over all features, so the one with the largest range decides who the neighbors are.
- ✗Scaling features for a decision tree and expecting a change — each split thresholds one feature, so a linear rescale leaves its predictions the same.
Follow-up questions
- →Which scaler would you pick when a feature has heavy outliers?
- →Why must the scaler be fitted on the training set only?
JuniorTheoryCommonWhat does this program print?
JuniorTheoryCommon
What does this program print?
0 1
Open full question →Common mistakes
- ✗Expecting
0 0: scaling changes the distances, so after it the age gap of 35 years outweighs the income gap of 6000, and the nearest point becomes[27, 46000], class1. - ✗Expecting
1without scaling: raw income differences are thousands and age differences are tens, so income alone picks[60, 40500], class0.
Follow-up questions
- →Why must the scaler be fitted on the training set only and then applied to the query point?
- →Would a decision tree's prediction change after the same scaling, and why?
JuniorTheoryCommonWhat does this program print?
JuniorTheoryCommon
What does this program print?
1 0
Open full question →Common mistakes
- ✗Expecting
1 1: the nearest point[3, 3]is class1, but the next two,[3, 0]and[1, 3], are class0, so three neighbors vote0. - ✗Expecting
0for one neighbor because class0has more training points: with one neighbor only the closest point counts, and[3, 3]at distance1.0is class1.
Follow-up questions
- →Which label would this code return with all five points as neighbors, and why?
- →How would weighting each vote by inverse distance change the answer for three neighbors?
