All topics
Dimensionality Reduction: LDA and Kernel PCA
Reducing features beyond plain PCA: supervised LDA and its component limit, and kernel PCA for non-linear structure.
Sort:
3 questions
JuniorTheoryCommonWhat can kernel PCA (principal component analysis) do that linear PCA cannot, and what do you lose?
JuniorTheoryCommon
What can kernel PCA (principal component analysis) do that linear PCA cannot, and what do you lose?
Kernel PCA runs PCA in an implicit kernel feature space, so it can unfold non-linear structure such as two concentric circles that linear PCA cannot separate. Its components are no longer linear combinations of the original features.
Common mistakes
- ✗Treating kernel PCA as a faster linear PCA — a non-linear kernel finds different components, and the kernel matrix grows with the square of the sample count.
- ✗Reading kernel PCA components like PCA loadings — they live in the implicit kernel space, so there is no weight per input feature to interpret.
Follow-up questions
- →Why does kernel PCA need a separate step to map points back to the input space?
- →How would you choose the RBF
gammafor kernel PCA?
JuniorTheoryCommonThe iris data has 150 flowers, 4 features and 3 classes; what does this program print?
JuniorTheoryCommon
The iris data has 150 flowers, 4 features and 3 classes; what does this program print?
(150, 2)
Open full question →Common mistakes
- ✗Expecting
(150, 4), as withPCA: LDA gives at most C − 1 components for C classes, so with 3 classes it keeps 2. - ✗Expecting one component per class,
(150, 3): the centers of 3 classes lie in a plane, so LDA finds only 2 separating directions.
Follow-up questions
- →What would
LinearDiscriminantAnalysis(n_components=3)do on the same data? - →How many components can LDA give for a binary classification task?
JuniorTheoryCommonHow does linear discriminant analysis (LDA) differ from principal component analysis (PCA)?
JuniorTheoryCommon
How does linear discriminant analysis (LDA) differ from principal component analysis (PCA)?
LDA is supervised: it uses the class labels to find directions that best separate the classes, and gives at most C − 1 components for C classes. PCA ignores labels and keeps the directions of maximum variance in the features.
Common mistakes
- ✗Treating LDA as an unsupervised PCA variant — it needs labels and optimizes class separation, not the variance of the features.
- ✗Asking LDA for more components than the classes allow — with C classes it yields at most C − 1, so a binary task gets one axis.
Follow-up questions
- →Why can PCA discard the one direction that separates two classes?
- →When would you run PCA before LDA?
