Underfitting and overfitting

Polynomial features are powerful. In Section 2 they bent a regression line into a curve, and in this section they bent a decision boundary into a circle. So why not always use the highest degree you can? Because there’s a catch, and it’s the central problem of machine learning. The goal is never to fit the training data; it’s to make good predictions on new data the model has never seen. That ability is called generalisation.

Three fits of the same data

Imagine twelve measurements of some smooth relationship, each with a little random noise, the way real measurements always have. We fit polynomials of different degrees.

• Degree 1, a straight line. It can’t follow the curve at all. It misses the training points badly, and it would miss new points just as badly. This is underfitting: the model is too simple for the pattern. It’s also called high bias, because the model has a strong built-in belief (that the world is a straight line) that the data can’t talk it out of.
• Degree 5 or so. It follows the overall shape and ignores the jitter. It’s a little off on most training points, but it would predict new points well. About right.
• Degree 12. With 13 parameters and 12 points it can pass through every training point exactly, so training error is zero. But between the points it swings wildly, because it’s fitting the noise as if it were signal. On new points it’s terrible. This is overfitting, also called high variance: train it on a slightly different sample and you’d get a completely different wiggle.

Filled dots are the training data the model learns from. Hollow dots are test data: fresh examples from the same process that the model never sees during training. Drag the degree from 1 to 12 and keep an eye on the right panel, which plots both errors for every degree.

Training error lies, test error tells the truth

The panel shows the classic pattern:

• Training error only ever goes down as the degree rises. A more flexible model can always fit the points it’s shown at least as well. Taken alone, training error would tell you to pick degree 12.
• Test error falls, then rises again, in a U shape. It’s lowest at a middle degree, around 5 here, and shoots up once the model starts memorising noise.

The gap between the two is the signature of overfitting. Small training error and large test error means the model has memorised rather than learned. Large error on both means it’s underfitting.

That’s why practitioners always keep some data back. Split the examples you have, train on one part, and measure on the other. You met the train/test split and data leakage in Python for Data, section 3. Section 5 refines this with a third, validation set used to choose things like the degree, so the test set stays a truly honest final exam.

import numpy as np

rng = np.random.default_rng(0)
def truth(x):
    return 0.8 * np.sin(2 * np.pi * x) + 0.3 * x

x_train = np.sort(rng.uniform(0, 1, 12)); y_train = truth(x_train) + rng.normal(0, 0.2, 12)
x_test  = rng.uniform(0, 1, 200);         y_test  = truth(x_test)  + rng.normal(0, 0.2, 200)

for degree in [1, 3, 5, 9]:
    coeffs = np.polyfit(x_train, y_train, degree)
    train_mse = np.mean((np.polyval(coeffs, x_train) - y_train) ** 2)
    test_mse = np.mean((np.polyval(coeffs, x_test) - y_test) ** 2)
    print(f"degree {degree}: training error {train_mse:.4f}   test error {test_mse:.4f}")

Your exact numbers depend on the random sample, but the shape of the story doesn’t: the training error keeps shrinking while the test error turns round and climbs.

Classifiers overfit too

Everything here applies to classification. In the decision-boundary widget, a degree-6 model on noisy data can draw a boundary that snakes around individual points to get every training example right, then misclassifies new ones. A boundary that is a little smoother and gets one or two training points wrong usually generalises better.

What can you do about overfitting?

1. Collect more data. With hundreds of points instead of twelve, even a flexible model can’t wiggle through all of them, and the noise averages out. Often the best fix, when it’s possible.
2. Use fewer features. Choose a lower degree, or drop features that don’t help. This works, but throws information away, and choosing which features to drop is itself a guess.
3. Regularisation. Keep all the features but discourage the model from using huge weights, which is what those wild swings need. This is the subject of the next lesson.

Check yourself