Feature engineering and polynomial regression

So far the features have been whatever columns came with the data. But the columns you’re handed aren’t always the most useful ones. Often you, the human, know something about the problem that suggests a better feature. Building those features is called feature engineering, and it’s one of the most valuable skills in practical machine learning.

Inventing a feature

Suppose a land registry gives you two columns for each plot: its frontage (width along the street) and its depth, both in metres. You could fit

But you probably suspect that what buyers pay for is the area. So create a third feature, $x_3 = \text{frontage} \times \text{depth}$, and let the model use all three:

Gradient descent then decides how much weight each one deserves. If area is what matters, $w_3$ will end up doing most of the work. You haven’t changed the algorithm at all, only what you feed it. The same trick covers ratios (price per square foot), differences (years since renovation), counts and flags (has a garage: 1 or 0).

Curves from straight lines

Here’s the most useful feature-engineering trick of all. Look at this data: fuel economy of cars against their speed. Cars are thirsty in slow traffic, most efficient at moderate speed and thirsty again at high speed because of air resistance. That’s a hump, and no straight line fits a hump.

The fix: from the single feature $x$ (speed), engineer new features $x^2$, $x^3$ and so on, and run ordinary multiple linear regression on them:

This is polynomial regression. The curve bends, yet to the algorithm it’s still linear regression: the model is a weighted sum of features, and it’s linear in the weights $w_j$. The cost, the gradient and the code from the last three lessons all work unchanged. The highest power used is the degree of the polynomial.

Explore the degree

Move the degree slider. The table under the plot lists the features the model receives. Then turn on Show new data to see cars the model never trained on, and compare the two errors in the readout. Play the walkthrough for a guided tour.

Three things to take away:

1. Degree 1 underfits. A line can’t bend, so it’s wrong in a systematic way: too high at the ends, too low in the middle. Adding data won’t help; the model is too simple.
2. Degree 2 or 3 fits well, both on the training cars and on the new ones.
3. Degree 10 overfits. It wiggles to pass close to every training car, so its training error is the lowest of all, but on new cars its error is more than ten times worse than degree 3. It has memorised the noise instead of learning the trend.

The lesson: a lower training cost does not mean a better model. You need to check on data the model hasn’t seen. The next section is about exactly this problem, overfitting, and the tools that tame it.

Scaling is not optional here

Look at the feature table again. Speed runs from about 10 to 80, speed squared up to about 6,000 and speed cubed up to nearly 500,000. That’s the feature-scaling problem from two lessons ago, in its most extreme form. Switch the widget to Gradient descent and untick Scale features: even after thousands of iterations the curve has hardly moved, because a learning rate small enough for $x^3$ is useless for $x$. Tick scaling again and the same descent finds the curve.

Let’s see it in code with ten cars. np.column_stack glues columns side by side into a matrix (Python for Data, section 1).

import numpy as np

speed = np.array([15, 22, 30, 38, 45, 52, 60, 67, 75, 80], dtype=float)
mpg = np.array([21.5, 25.0, 28.1, 29.6, 30.4, 30.2, 28.9, 27.0, 23.8, 21.9])

def train(F, y, alpha, iters):
    m, n = F.shape
    w, b = np.zeros(n), 0.0
    for _ in range(iters):
        err = F @ w + b - y
        w = w - alpha * (F.T @ err) / m
        b = b - alpha * err.mean()
    return w, b, ((F @ w + b - y) ** 2).mean() / 2

for degree in [1, 2]:
    F = np.column_stack([speed ** k for k in range(1, degree + 1)])   # x, x^2, ...
    F_norm = (F - F.mean(axis=0)) / F.std(axis=0)                     # z-score every column
    w, b, cost = train(F_norm, mpg, alpha=0.5, iters=2000)
    print(f"degree {degree}: cost {cost:.3f}")

The quadratic’s cost is hundreds of times lower than the line’s. Now the same degree-2 model without scaling. The largest safe learning rate for unscaled $x^2$ here is tiny:

F_raw = np.column_stack([speed, speed ** 2])
w, b, cost = train(F_raw, mpg, alpha=1e-7, iters=2000)
print(f"unscaled degree 2: cost {cost:.3f}")
print("predictions:", np.round(F_raw @ w + b, 1))

Its predictions don’t even trace the hump. Scale your polynomial features, every time.

Your turn

import numpy as np

speed = np.array([15, 22, 30, 38, 45, 52, 60, 67, 75, 80], dtype=float)

def poly_features(x, degree):
    # your code here: columns x, x**2, ..., x**degree
    return None

def poly_scaled(x, degree):
    # your code here
    return None

print(poly_features(speed, 3))

Check yourself