JuniorTheoryCommon

What does this program print?

Python

from sklearn.feature_extraction.text import TfidfVectorizer

texts = ["the good day", "the bad day", "the good night"]
vec = TfidfVectorizer().fit(texts)
idf = vec.idf_
voc = vec.vocabulary_
print(round(idf[voc["the"]], 2), round(idf[voc["bad"]], 2))

1.0 1.69

Expecting 0.0 1.69: scikit-learn adds 1 to the logarithm, so a word found in every text gets the smallest weight, 1.0, not zero.
Expecting 1.0 3.0: IDF is the logarithm ln((1 + n) / (1 + df)) + 1, not the plain ratio of texts n / df, so bad gets ln(2) + 1.

Why does TF-IDF scale every row to length 1 after weighting the counts?
Which words would the stop-word option stop_words="english" remove, and how does that differ from a low IDF?

Back to questions Open topic in roadmap →Answer it yourself — Quiz

Back to questions

JuniorTheoryCommon

What does this program print?

Python

from sklearn.feature_extraction.text import TfidfVectorizer

texts = ["the good day", "the bad day", "the good night"]
vec = TfidfVectorizer().fit(texts)
idf = vec.idf_
voc = vec.vocabulary_
print(round(idf[voc["the"]], 2), round(idf[voc["bad"]], 2))

1.0 1.69

Expecting 0.0 1.69: scikit-learn adds 1 to the logarithm, so a word found in every text gets the smallest weight, 1.0, not zero.
Expecting 1.0 3.0: IDF is the logarithm ln((1 + n) / (1 + df)) + 1, not the plain ratio of texts n / df, so bad gets ln(2) + 1.

Why does TF-IDF scale every row to length 1 after weighting the counts?
Which words would the stop-word option stop_words="english" remove, and how does that differ from a low IDF?

Back to questions Open topic in roadmap →Answer it yourself — Quiz