Feature scaling

The last lesson ended with a puzzle. On the house table, gradient descent learnt the size weight but left the bedroom and floor weights at almost zero, and any learning rate big enough to move them made the size weight blow up. The cause was the scales of the features: size runs into the thousands, bedrooms from 1 to 5. The cure is to rescale the features so they live on similar ranges. It takes two lines of NumPy and can make training hundreds of times faster.

Why scale changes the landscape

Think about what a weight does. A change of 0.01 in the size weight changes a 2,000 sq ft house’s prediction by 20. The same change in the bedrooms weight changes it by 0.03. So the cost is extremely sensitive to the size weight and almost indifferent to the bedrooms weight.

On a contour map of $J$ over the two weights, sensitivity shows up as how tightly packed the contour lines are. Very sensitive in one direction and lazy in the other means the contours are long, thin ellipses: a narrow canyon with steep walls and a nearly flat floor.

Gradient descent always steps straight downhill, which is across the contour lines. In a narrow canyon “straight downhill” points mostly at the opposite wall, not along the floor toward the minimum. So descent bounces from wall to wall, a zig-zag, while creeping slowly along the floor. Make the steps bigger to creep faster and the bounces grow until they fly out of the canyon: divergence. You’re stuck with a learning rate set by the steepest direction and progress set by the flattest.

If all features have similar ranges, the cost is about equally sensitive to every weight, the contours become near circles, and “straight downhill” points right at the minimum.

See it

Below are two features, house size (in hundreds of square feet, from about 7 to 24) and bedrooms (1 to 5), predicting price. The left panel is the contour map of $J(w_1, w_2)$; to draw it flat, $b$ is held at its best value. Descent starts at $w_1 = w_2 = 0$. The right panel is the cost after each step. Switch between Raw features, Mean normalisation and Z-score, and press Run each time.

With raw features the contours are stretched about 12 to 1, the path zig-zags, and it needs around 190 steps to converge. After scaling, the map is nearly round and descent arrives in about 7 steps, with the same rule for choosing $\alpha$. Try nudging $\alpha$ up in raw mode to make it faster: you’ll hit the divergence limit long before the zig-zag goes away.

Two common ways to scale

Mean normalisation centres each feature on zero and divides by its range:

where $\mu_j$ is the mean of feature $j$ over the training set. The result lies roughly between $-0.5$ and $0.5$.

Z-score normalisation (also called standardisation) centres each feature and divides by its standard deviation $\sigma_j$, which you met in Math for ML, section 4:

Now every feature has mean 0 and standard deviation 1, so most values fall between about $-3$ and $3$. Z-scoring is the default choice in practice because a single extreme value can’t squash everything else the way it can stretch the range.

A useful rule of thumb: values roughly between $-1$ and $1$ are ideal, anything up to about $\pm 3$ is fine. A feature that runs from 0 to 3,000 or from 0.001 to 0.002 is worth rescaling. Rescaling never hurts, so when in doubt, scale.

Scale the house table

Back to the six houses from earlier lessons. NumPy computes the mean and standard deviation of every column at once with axis=0 (Python for Data, section 1):

import numpy as np

X = np.array([
    [1850, 3, 2, 12],
    [1200, 2, 1, 40],
    [2400, 4, 2, 8],
    [950, 2, 1, 55],
    [1600, 3, 1, 25],
    [3100, 5, 2, 3],
], dtype=float)
y = np.array([412, 236, 538, 178, 330, 690], dtype=float)

mu = X.mean(axis=0)               # one mean per feature
sigma = X.std(axis=0)             # one standard deviation per feature
X_norm = (X - mu) / sigma         # broadcasting does every column at once

print("means: ", np.round(mu, 1))
print("sigmas:", np.round(sigma, 1))
print("scaled table:\n", np.round(X_norm, 2))
print("new means:", np.round(X_norm.mean(axis=0), 6), " new sigmas:", X_norm.std(axis=0))

Now train on the scaled features. Last time we were stuck with $\alpha = 10^{-7}$; now a learning rate of 0.5 is safe.

def gradient_descent(X, y, alpha, iters):
    m, n = X.shape
    w, b = np.zeros(n), 0.0
    for _ in range(iters):
        err = X @ w + b - y
        w = w - alpha * (X.T @ err) / m
        b = b - alpha * err.mean()
    cost = ((X @ w + b - y) ** 2).mean() / 2
    return w, b, cost

w, b, cost = gradient_descent(X_norm, y, alpha=0.5, iters=1000)
print("weights:", np.round(w, 2), " b:", round(b, 2))
print("final cost:", round(cost, 3), "(unscaled, after 1000 steps, it was about 175)")

Every weight now moves, and the cost after the same thousand steps is hundreds of times lower. Notice that $b$ came out as the average price: with centred features, the “base price” is the price of an average house.

new_house = np.array([2000, 3, 2, 20], dtype=float)
new_scaled = (new_house - mu) / sigma          # the TRAINING mu and sigma
print("predicted price: $", round(float(new_scaled @ w + b)), "thousand")

Your turn

import numpy as np

X = np.array([
    [1850, 3, 2, 12],
    [1200, 2, 1, 40],
    [2400, 4, 2, 8],
    [950, 2, 1, 55],
    [1600, 3, 1, 25],
    [3100, 5, 2, 3],
], dtype=float)
y = np.array([412, 236, 538, 178, 330, 690], dtype=float)

def zscore_normalize(X):
    # your code here
    mu = None
    sigma = None
    X_norm = None
    return X_norm, mu, sigma

def scale_new(x_new, mu, sigma):
    # your code here
    return None

X_norm, mu, sigma = zscore_normalize(X)
print(X_norm)

Check yourself