The cost function: how wrong is the line?

You can judge a line by eye, but a computer needs a number: one score that is small when the line fits well and large when it fits badly. That score is called the cost function, and it’s the compass that every learning algorithm in this course steers by.

Step 1: the error for one house

Take one training example $(x^{(i)}, y^{(i)})$. The model predicts $\hat{y}^{(i)} = f_{w,b}(x^{(i)})$. The error for that house is how far the prediction is from the truth:

$$\hat{y}^{(i)} - y^{(i)}$$

On the chart it’s the vertical gap between the dot and the line. Positive means the model guessed too high; negative means too low.

Step 2: square it

We can’t just add up the errors, because a guess that’s $50k too high and one that’s $50k too low would cancel out and look perfect. So we square each error. That makes every error positive, and it punishes big misses much more than small ones: an error of 2 counts four times as much as an error of 1.

Squaring has a lovely picture: the squared error is literally the area of a square whose side is the gap between the dot and the line.

Step 3: average over all houses

Add up the squares for all $m$ houses and divide by $m$ to get the average, so the score doesn’t grow just because we collected more data. By convention we divide by $2m$ instead of $m$; the extra 2 makes a later formula tidier and doesn’t change which line is best.

Putting it together, the squared-error cost function is:

$$J(w,b) = \frac{1}{2m} \sum_{i=1}^{m} \left( f_{w,b}(x^{(i)}) - y^{(i)} \right)^2$$

Read it out loud: “for every example, take the prediction minus the true value, square it, add them all up, and divide by two m.”

Watch the squares

Here are six houses, so every square is visible. Move $w$ and $b$ and watch the squares grow and shrink; the readout adds up their areas to give $J$. On the right, the curve shows $J$ for every possible $w$ (with $b$ held where it is), and the dot shows where your current line sits on it.

Loading the interactive graphic…

Play walkthroughStep 1 of 9

Here are six houses and a line that's clearly too flat. Each vertical gap is one prediction's error.

Try to make the total area as small as you can. You will find there is one best slope: tilt the line less or more and the squares get bigger. That best point is the bottom of the curve on the right.

The goal of training

We now have a precise goal. Out of all possible lines, find the parameters that make the cost as small as possible:

$$\min_{w,b} \; J(w,b)$$

In the next lesson you’ll see what $J$ looks like when both $w$ and $b$ can change, a whole landscape, and after that, how a computer finds its lowest point automatically.

Check yourself

Self-check · 4 questions

The cost function

1. Why are errors squared before adding them up?

Because the line is straightTo make the cost negativeTo make the numbers smallerSo positive and negative errors can't cancel out, and big errors count more

2. Three houses have errors $\hat{y} - y$ of 1, −1 and 2. What is $J$ using $\frac{1}{2m}\sum(\hat{y}-y)^2$?

2/31/331

3. What does a cost of $J = 0$ mean?

The learning rate is zeroThere are no training examples leftThe model is uselessThe line passes exactly through every training example

4. What is the goal of training a linear regression model?

Find the $w$ and $b$ that make $J(w,b)$ as small as possibleMake $J(w,b)$ equal to $m$Pass the line through the first and last pointsMake $w$ as large as possible

Check answers