Is gradient descent working?

With one or two parameters you could watch gradient descent roll down a picture of the cost. With four features, or four million parameters, there’s no picture to watch. So how do you know training is going well? You keep a record of the cost after every step and look at that instead. This one plot, the learning curve, is the first thing ML engineers check when a model misbehaves.

The learning curve

Plot the cost $J$ on the vertical axis against the iteration number on the horizontal axis. For a healthy run:

• the cost goes down after every single iteration, and
• it drops steeply at first, then levels off as the steps shrink near the minimum.

When the curve has flattened out, more iterations won’t buy much: gradient descent has converged. How many iterations that takes varies enormously between problems, from a few dozen to millions, which is exactly why you look rather than guess.

Here is a learning curve for the six-house table from earlier lessons, with z-scored features.

def train(X, y, alpha, iters):
    m, n = X.shape
    w, b = np.zeros(n), 0.0
    history = []
    for _ in range(iters):
        err = X @ w + b - y
        w = w - alpha * (X.T @ err) / m
        b = b - alpha * err.mean()
        history.append(((X @ w + b - y) ** 2).mean() / 2)
    return w, b, history

w, b, history = train(X, y, alpha=0.1, iters=100)
bpl.plot(range(1, 101), history, kind="line", title="Learning curve, alpha = 0.1")
print("cost after 1, 10 and 100 steps:", [round(float(history[k]), 1) for k in (0, 9, 99)])

(A hidden setup cell on this page loads and z-scores the data before anything else runs. The bpl.plot function draws charts on this site; you met it in Python for Data, section 3.)

Declaring convergence automatically

You can also let the code decide. A common automatic convergence test: pick a small number $\varepsilon$ (epsilon), say $10^{-3}$, and stop when one iteration lowers the cost by less than $\varepsilon$. It’s handy, but choosing $\varepsilon$ is fiddly (a good value depends on the scale of your costs), so it’s wise to glance at the curve as well.

What the curve tells you about α

The shape of the learning curve is a diagnosis. The widget runs gradient descent on two scaled house features; the right panel is its learning curve. Play the walkthrough, then try learning rates yourself with the α slider.

• Falls quickly, then flattens. A good learning rate.
• Falls steadily but slowly, never flattening in reasonable time. $\alpha$ is too small. Every step helps, there just aren’t big enough steps.
• Goes up, or bounces up and down. $\alpha$ is too large and the steps overshoot. Or there’s a bug, such as a plus where the update needs a minus.

That last point gives you a debugging trick. With a small enough $\alpha$, the cost of linear regression is guaranteed to fall on every iteration. So if you set $\alpha$ tiny and the cost still rises, the learning rate isn’t the problem: look for a mistake in the code.

A recipe for choosing α

Try a ladder of values, each about three times the last:

Run a modest number of iterations for each and compare the curves. Find the smallest value that’s clearly too small and the largest that still falls smoothly, then pick the largest one that behaves (or something just below it). Let’s run the ladder on the house data:

for alpha in [0.01, 0.03, 0.1, 0.3, 0.6]:
    _, _, h = train(X, y, alpha, iters=100)
    trend = "rising!" if h[-1] > h[0] else "falling"
    print(f"alpha = {alpha:<5} cost after 100 steps: {h[-1]:>14,.2f}   ({trend})")

_, _, h = train(X, y, alpha=0.6, iters=30)
bpl.plot(range(1, 31), h, kind="line", title="alpha = 0.6: the cost climbs")

For this data, 0.3 is the winner, and 0.6 is past the edge: the cost grows on every step. Feature scaling from the last lesson is what makes a single ladder like this work for every weight at once.

Your turn

import numpy as np

def converged_at(history, epsilon):
    # your code here
    return None

def is_diverging(history):
    # your code here
    return None

print(converged_at([10, 5, 3, 2.5, 2.4, 2.39], 0.05))
print(is_diverging([1, 2, 4, 8]))

Check yourself