Choosing the learning rate

Gradient descent has one knob you have to set yourself: the learning rate $\alpha$. It looks innocent, just a small number multiplying the slope, but it decides whether training takes a few steps, a few million, or never finishes at all.

Four ways it can go

Using the same update $w := w - \alpha \frac{\partial J}{\partial w}$:

• Too small: each step is tiny. Training does head downhill, but painfully slowly; you may run out of patience (or compute budget) long before reaching the bottom.
• About right: big confident steps at first, smaller ones as the slope flattens, and the cost settles at the minimum in a handful of iterations.
• Too large: each step jumps past the bottom to the other side of the bowl. It still zig-zags its way in, but it wastes steps bouncing back and forth.
• Far too large: each jump lands higher up the other side than where it started. The slope there is steeper, so the next jump is even bigger. The cost explodes instead of shrinking. This is called divergence.

Why can overshooting make things worse? Because the step size is $\alpha$ times the slope. Land somewhere steeper and your next step is bigger, which lands you somewhere steeper still.

Try every behaviour

Pick a preset, or move the $\alpha$ slider yourself, then press Run. The left chart shows the path on the cost curve; the right one shows the cost after every iteration, which is the chart practitioners actually watch.

Loading the interactive graphic…

Play walkthroughStep 1 of 8

We'll run gradient descent from the same starting point, with four different learning rates.

Reading the cost curve

When you train a real model you can’t see the landscape, but you can always plot $J$ after every iteration. That plot is your main diagnostic:

• Going down and flattening out: training is working and has converged.
• Going down, but slowly and steadily: $\alpha$ is probably too small; try a larger one.
• Going up, or jumping up and down: something is wrong. Most often $\alpha$ is too large, though a bug in the code can look the same. A handy check: with a very small $\alpha$ the cost should decrease on every single iteration. If it doesn’t, suspect a bug.

A practical recipe

There’s no formula that gives the perfect $\alpha$ for every problem, so practitioners try a few values spaced roughly three times apart, such as

$$0.001,\; 0.003,\; 0.01,\; 0.03,\; 0.1,\; 0.3,\; 1$$

run a short training for each, and pick the largest one whose cost goes down quickly and steadily, perhaps a bit smaller to be safe.

Check yourself

Self-check · 3 questions

Choosing the learning rate

1. Your cost decreases on every iteration, but very slowly. What is the most likely fix?

Stop trainingUse a larger learning rateUse a smaller learning rateAdd more parameters

2. The cost grows larger with every iteration. What is happening?

The data is too smallThe model is overfittingTraining has convergedGradient descent is diverging, most likely because alpha is too large

3. Which set of learning rates is a sensible first search?

0.001, 0.003, 0.01, 0.03, 0.1, 0.31, 10, 100Only 0.50.1, 0.2, 0.3, 0.4

Check answers