JuniorTheoryCommon

What does this program print?

Python

text = "Good film. Good, good cast!"
tokens = text.lower().split()
print(len(tokens), len(set(tokens)))

5 4

Expecting 5 3: split() cuts only at whitespace, so film. and good, keep their punctuation and differ from good.
Expecting 5 5: lower() runs before split(), so the first Good becomes good and matches the other good.

How would you clean the text so that every form of good becomes one token?
Why can deleting punctuation instead of replacing it with a space merge two words?

JuniorTheoryCommon

Python

text = "Good film. Good, good cast!"
tokens = text.lower().split()
print(len(tokens), len(set(tokens)))

5 4

Expecting 5 3: split() cuts only at whitespace, so film. and good, keep their punctuation and differ from good.
Expecting 5 5: lower() runs before split(), so the first Good becomes good and matches the other good.

How would you clean the text so that every form of good becomes one token?
Why can deleting punctuation instead of replacing it with a space merge two words?