Training, testing and prediction

A model is trained for its answers on new data: on emails that arrive tomorrow, on apartments not yet sold. There are no correct answers for them yet, so they cannot be used to check the model. That is why a model's quality is checked in advance, on examples with a known target that the model has not seen.

Five steps

Work on a model almost always follows one order:

• Collect a table of examples and split it into features X and a target y.
• Set aside some rows, usually a fifth to a quarter. This is the test set; the remaining rows are the training set.
• Train the model on the training set only.
• Get the model's answers on the test set and compare them with the known target.
• If the quality is good enough, predict the target for new samples.

The model does not see the test rows during training. To the model they are as new as tomorrow's emails, but their correct answers are known. That is why a score on them shows what to expect from the model later.

fit and predict

In the scikit-learn library, every model that predicts a target has the same two methods. The fit(X_train, y_train) method trains the model. It picks the parameters from rows with a known target and stores them inside the model. The predict(X_new) method applies the stored parameters to new rows and returns the predicted target.

The predict method does not need the target: that is what it outputs. predict itself learns nothing, however many times it is called. The model changes only when fit is called again.

Common mistake

Quality is checked on the same rows the model was trained on. Here is an extreme case—a model that memorizes the examples:

memory = {30: 3.1, 45: 4.4, 60: 6.2, 80: 7.9}

def predict(area):
    return memory.get(area, 0.0)  # ✗

print(predict(45))  # 4.4
print(predict(50))  # 0.0

On all four examples, this model never errs. But for a 50 m² apartment, which is not among the examples, it answers 0.0.

This is overfitting: the model learned the examples instead of the pattern. A score on the training rows cannot tell it from a good model. A score on the test set tells them apart at once.