JuniorTheoryCommon

What does this program print?

Python

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()
vec.fit(["red apple", "green apple"])
X = vec.transform(["Red red pear!"])
print(X.shape[1], X.sum())

3 2

Expecting 4 3: transform keeps the vocabulary learned by fit, so pear gets no column and is dropped.
Expecting 3 1: CountVectorizer lowercases text by default, so both Red and red are counted in the red column.

What goes wrong if you call fit_transform on the new texts instead of transform?
Why does CountVectorizer return a sparse matrix rather than a numpy array?

Back to questions Open topic in roadmap →Answer it yourself — Quiz

Back to questions

JuniorTheoryCommon

What does this program print?

Python

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()
vec.fit(["red apple", "green apple"])
X = vec.transform(["Red red pear!"])
print(X.shape[1], X.sum())

3 2

Expecting 4 3: transform keeps the vocabulary learned by fit, so pear gets no column and is dropped.
Expecting 3 1: CountVectorizer lowercases text by default, so both Red and red are counted in the red column.

What goes wrong if you call fit_transform on the new texts instead of transform?
Why does CountVectorizer return a sparse matrix rather than a numpy array?

Back to questions Open topic in roadmap →Answer it yourself — Quiz