Gradient descent: walking downhill — Machine Learning, Step by Step

In this lesson you will

Explain gradient descent as repeatedly stepping in the downhill direction of the cost
Read the update rule w := w − α · dJ/dw and say what each part does
Explain why the steps shrink automatically near the minimum
State the update rules for both w and b in linear regression

You now know what we want: the $w$ and $b$ at the bottom of the cost bowl. With two parameters you can find it by eye. With a million you can’t, and you can’t try every combination either. We need a method that finds the bottom without seeing the whole landscape. That method is gradient descent, and it trains nearly every model in this course, from this straight line up to large neural networks.

The idea: lost on a foggy hill

Imagine standing on a hillside in thick fog. You can’t see the valley, but you can feel the ground under your feet. A sensible strategy: feel which way slopes down most steeply, take a small step that way, and repeat. Eventually you reach the bottom.

Gradient descent does exactly this on the cost landscape:

Start with some guess for the parameters (often just $w = 0$ , $b = 0$ ).
Measure the slope of the cost at that point.
Take a small step downhill.
Repeat until the steps become tiny: you’ve arrived.

The update rule

To keep things simple, hold $b$ fixed for a moment and look at $J$ as a curve in $w$ . The slope of that curve at the current $w$ is the derivative $\frac{\partial J}{\partial w}$ . You don’t need calculus to follow it: it’s just “how much $J$ changes if you nudge $w$ a little”. One step of gradient descent is:

w := w - \alpha \frac{\partial J}{\partial w}

The symbol $:=$ means “replace $w$ with”. Each part has a job:

The slope tells you the direction. If the slope is negative (the curve goes down to the right), subtracting it increases $w$ , moving right, downhill. If it’s positive, $w$ decreases. Either way you head downhill.
$\alpha$ (alpha), the learning rate, is a small positive number like $0.1$ that controls how big each step is. The next lesson is all about choosing it.

Here’s a neat bonus: near the bottom the curve flattens, so the slope gets close to zero and the steps shrink automatically, even though $\alpha$ stays the same. Gradient descent slows down as it arrives.

Roll the ball

The ball sits on $J(w)$ for our six houses, with $b$ fixed at $0.5$ . The straight line touching the curve is the tangent: its steepness is the slope. Press Step to take one update and watch the numbers in the update rule change; press Run to keep going. The small chart shows the line through the data improving as the ball rolls.

Try starting the ball on the right side of the bowl (drag the start- $w$ slider past the bottom). The slope is now positive, so the same rule moves $w$ to the left. One rule works from both sides.

Both parameters at once

For the full model we update $w$ and $b$ together, each using its own slope. For the squared-error cost these slopes work out to:

\frac{\partial J}{\partial w} = \frac{1}{m} \sum_{i=1}^{m} \left( f_{w,b}(x^{(i)}) - y^{(i)} \right) x^{(i)} \qquad \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^{m} \left( f_{w,b}(x^{(i)}) - y^{(i)} \right)

(This is where the $\frac{1}{2}$ in the cost paid off: it cancelled the 2 that comes down from the square.) Then, repeating until convergence:

w := w - \alpha \frac{\partial J}{\partial w} \qquad b := b - \alpha \frac{\partial J}{\partial b}

One important detail: compute both slopes using the current $w$ and $b$ , and only then update both. This is called a simultaneous update. If you updated $w$ first and used the new $w$ to compute $b$ ‘s slope, you’d be mixing two different points on the landscape.

Because each step uses all $m$ training examples, this version is called batch gradient descent. Later you’ll meet faster variants that use a handful of examples per step.