Regularisation

The degree-12 curve in the last lesson passed through every training point and swung wildly in between. Look at its weights and you’ll find enormous numbers, hundreds or thousands, pulling in opposite directions and almost cancelling at the training points. Those huge, finely balanced weights are what make the wiggles possible.

That suggests a gentler fix than throwing features away: keep all the features, but make large weights expensive. That’s regularisation.

The regularised cost

Add a penalty to the cost that grows with the size of the weights:

This is regularised linear regression. The new number $\lambda$ (lambda) is the regularisation parameter, and it sets the exchange rate between the two goals:

• The first term wants to fit the training data as closely as possible.
• The second term wants every $w_j$ near zero.

Minimising the sum forces a compromise. A weight is only allowed to be large if it buys a real improvement in the fit. Tiny wiggles that chase the noise in one or two points aren’t worth the price, so they disappear. Dividing by $2m$, like the first term, keeps a sensible $\lambda$ from depending on how much data you have.

Two conventions to notice:

• The sum runs over $w_1$ to $w_n$, not $b$. The bias only shifts the curve up or down; it doesn’t create wiggles, so there’s no reason to shrink it. (Penalising it makes almost no difference in practice.)
• This squared-weight penalty is called L2 regularisation, or ridge for linear regression. In neural networks the same idea is usually called weight decay, for a reason you’ll see in a moment.

Choosing λ

Start with degree 12 and $\lambda = 0$: perfect on the training points, wild in between, with a largest weight in the hundreds. Now raise $\lambda$ one notch at a time.

• $\lambda = 0$: no regularisation. Overfits.
• Moderate $\lambda$ (0.1 is the sweet spot here): the weights collapse to modest sizes, the curve smooths out and follows the true shape, and test error drops from about 8 to under 0.1. The model keeps its 12 degrees of freedom but no longer abuses them.
• Huge $\lambda$ (10): the penalty dominates. Every weight is squashed nearly to zero, the curve flattens towards a horizontal line at the average $y$, and the model underfits.

So $\lambda$ is a dial between overfitting and underfitting. It’s a hyperparameter: a setting you choose, not a parameter the model learns. You pick it the same way you’d pick the degree, by trying several values and keeping the one with the lowest error on held-out data.

Regularised logistic regression

The same penalty works for any model. For logistic regression, add it to the log loss:

where $f^{(i)}$ is short for $f_{\vec{w},b}(\vec{x}^{(i)})$. On a high-degree decision boundary, regularisation smooths the snaking boundary into a simpler curve that generalises better. It also fixes the problem from the gradient-descent lesson: on perfectly separable data the weights no longer grow forever, because the penalty eventually outweighs the shrinking log loss.

Gradient descent with regularisation

The derivative of $\frac{\lambda}{2m} w_j^2$ with respect to $w_j$ is $\frac{\lambda}{m} w_j$, so each weight’s gradient gains one term. For linear and logistic regression alike:

The gradient for $b$ is unchanged. Rearranging the update for $w_j$ shows what the extra term does:

The second part is the usual update. The first part multiplies $w_j$ by a number just below 1, for example $1 - 0.1 \times 1/50 = 0.998$, on every iteration. Each step, every weight decays a little towards zero, and only the pull of the data holds it up. Hence “weight decay”.

Your turn

Write the regularised logistic cost, then its gradient. You can reuse the formulas above directly; the only new parts are the penalty and its derivative.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def compute_cost_reg(X, y, w, b, lambda_=1.0):
    m = len(y)
    f = sigmoid(X @ w + b)
    log_loss = -np.mean(y * np.log(f) + (1 - y) * np.log(1 - f))
    penalty = 0.0  # replace: lambda_ / (2m) times the sum of squared weights
    return log_loss + penalty

X = np.array([[0.5, 1.5], [1.0, 1.0], [1.5, 0.5], [3.0, 0.5], [2.0, 2.0], [1.0, 2.5]])
y = np.array([0, 0, 0, 1, 1, 1])
print(compute_cost_reg(X, y, np.array([1.0, 1.0]), -3.0, lambda_=0.7))   # about 0.4835

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def compute_gradient_reg(X, y, w, b, lambda_=1.0):
    m = len(y)
    err = sigmoid(X @ w + b) - y
    dj_dw = X.T @ err / m
    dj_db = np.mean(err)
    # add the regularisation term to dj_dw here
    return dj_dw, dj_db

X = np.array([[0.5, 1.5], [1.0, 1.0], [1.5, 0.5], [3.0, 0.5], [2.0, 2.0], [1.0, 2.5]])
y = np.array([0, 0, 0, 1, 1, 1])
print(compute_gradient_reg(X, y, np.array([0.5, -0.5]), 0.3, lambda_=0.7))

Check yourself