All topics
Naive Bayes
Probabilistic classifiers built on Bayes' theorem: the independence assumption, Gaussian versus multinomial variants, and Laplace smoothing.
Sort:
3 questions
JuniorTheoryCommonWhat does this program print?
JuniorTheoryCommon
What does this program print?
0.31
Open full question →Common mistakes
- ✗Expecting
0.8: that isP(word | spam); the code computesP(spam | word), and since spam is only 10% of mail, most emails with the word are normal. - ✗Expecting
0.08: that numerator isP(word | spam) · P(spam); dividing byspam + ham, which isP(word) = 0.26, turns it into a probability.
Follow-up questions
- →How would the answer change if spam made up half of all mail?
- →Why can a classifier skip the division by
P(word)when it only picks the class?
JuniorTheoryCommonWhen do you use GaussianNB and when MultinomialNB in scikit-learn?
JuniorTheoryCommon
When do you use GaussianNB and when MultinomialNB in scikit-learn?
GaussianNB models each continuous feature as a normal distribution within each class, for inputs such as measurements. MultinomialNB models non-negative counts such as word counts, which makes it the usual baseline for text.
Common mistakes
- ✗Treating Naive Bayes variants as interchangeable — each assumes a different likelihood, so Gaussian on word counts or multinomial on raw prices models the data wrongly.
- ✗Reading «multinomial» as «multi-class» — it names the count distribution of the features, and both variants handle any number of classes.
Follow-up questions
- →Which Naive Bayes variant fits binary word-presence features?
- →Why does
MultinomialNBreject negative feature values?
JuniorTheoryCommonWhat goes wrong in multinomial Naive Bayes without additive smoothing, and how does alpha fix it?
JuniorTheoryCommon
What goes wrong in multinomial Naive Bayes without additive smoothing, and how does alpha fix it?
If a vocabulary word never appeared in a class's training documents, its likelihood for that class is zero, and the product zeroes that class for any document with the word. Additive smoothing adds alpha to every count, keeping it positive.
Common mistakes
- ✗Assuming a word unseen with a class is skipped — a vocabulary word with a zero count gives a zero likelihood that wipes out that class's whole score.
- ✗Confusing a word unseen with one class and an out-of-vocabulary word — a fitted vectorizer drops the latter; smoothing is for zeros inside the vocabulary.
Follow-up questions
- →What happens to the predictions as
alphagrows very large? - →Why do implementations sum log-probabilities instead of multiplying probabilities?
