Rules versus examples

An email service has to tell spam from ordinary mail. A programmer can write a rule by hand: an email with the word "win" in the subject is spam.

def is_spam(subject):
    return "win" in subject.lower()

print(is_spam("You WIN a prize"))  # True
print(is_spam("Winter sale starts today"))  # True
print(is_spam("Free iPhone for you"))  # False

The rule fails in both directions. The sale email landed in spam because of the letters "win" inside the word "Winter". The scam email about an iPhone got through because it lacks the word.

You can keep adding exceptions, but spammers change their wording every week. The rules grow into thousands, and they start to contradict each other.

Learning from examples

Machine learning solves the problem differently. The program gets examples with correct answers instead of rules: thousands of emails, each marked as spam or not spam. It finds the rule on its own. When spammers change their wording, the model is trained again on fresh examples, and the rule updates without a programmer.

A model is a function that takes input data and returns an answer. Its behavior is set by numbers—the model parameters. Training picks the parameters so that the model makes as few errors on the examples as possible.

Picking a parameter

Suppose a model predicts an apartment's price in millions from its area: the area is multiplied by a parameter w. The examples are four apartments with known prices. Training tries several values of w and computes the total error on the examples for each:

areas = [30, 45, 60, 80]
prices = [3.1, 4.4, 6.2, 7.9]

def total_error(w):
    error = 0
    for area, price in zip(areas, prices):
        error += abs(w * area - price)
    return error

print(round(total_error(0.05), 2))  # 10.85
print(round(total_error(0.1), 2))  # 0.5
print(round(total_error(0.15), 2))  # 10.65

The total_error function adds up the misses as absolute values, so errors up and down do not cancel out. The error is smallest at w = 0.1, and that value becomes the parameter of the trained model.

For an apartment of 50 m², the model predicts 0.1 * 50, that is, 5 million. No such apartment was among the examples. The model answers on new data using a rule it found on old data, and that is its whole purpose.

Real models are more complex. They can have thousands of parameters, which are found by fast methods rather than by trying values one by one. The scheme stays the same: examples with answers, the error on them, and parameters that keep the error small.