The log loss: why not squared error?

Squared error plus a sigmoid makes a bumpy, flat-bottomed cost that gradient descent can stall on. The log loss charges confident mistakes heavily and gives one smooth bowl.

Lesson · 16 min

In this lesson you will

Explain why squared error is a poor cost for logistic regression
Write the loss for one example for y = 1 and for y = 0, and combine them into one formula
Compute the logistic cost J(w, b) over a training set in NumPy
Connect the log loss to maximum likelihood

We have a model, $f_{\vec{w},b}(\vec{x}) = g(\vec{w} \cdot \vec{x} + b)$ . To train it we need a cost function: one number that says how wrong the model is on the training set, which gradient descent can then push downhill. The obvious first try is the cost that served us so well for linear regression.

Squared error, again?

Squared error would be

J(\vec{w}, b) = \frac{1}{m} \sum_{i=1}^{m} \frac{1}{2} \left( f_{\vec{w},b}(\vec{x}^{(i)}) - y^{(i)} \right)^2.

For linear regression this was a smooth bowl with one bottom, perfect for gradient descent. But now $f$ goes through a sigmoid, and that changes the shape of $J$ completely. Two things go wrong:

The cost has flat plateaus. When the model is very wrong and very confident, the sigmoid is flat out there, so nudging $w$ barely changes the prediction, and the squared error barely changes either. The slope is almost zero and gradient descent stalls, even though the model is terrible.
The surface isn’t convex. It bends the wrong way in places, so it isn’t one clean bowl. Gradient descent can get stuck or crawl.

There’s also a common-sense problem. Suppose a tumour is malignant ( $y = 1$ ) and the model says 0.001, “almost certainly benign”. Squared error charges $\frac{1}{2}(0.001 - 1)^2 \approx 0.5$ . That’s the most it can ever charge, for the worst possible answer. A confidently wrong medical prediction deserves a far bigger penalty.

A loss built for probabilities

The fix is to define the loss for one example, $L$ , separately for each label:

L\big(f, y\big) = \begin{cases} -\log(f) & \text{if } y = 1 \\ -\log(1 - f) & \text{if } y = 0 \end{cases}

where $\log$ is the natural logarithm. Read the $y = 1$ case. If the model says $f = 1$ , the loss is $-\log 1 = 0$ : perfect, no charge. As $f$ drops towards 0 the loss climbs without limit: $-\log 0.5 \approx 0.69$ , $-\log 0.1 \approx 2.3$ , $-\log 0.001 \approx 6.9$ . Being confidently wrong is very expensive. The $y = 0$ case is the mirror image: no charge at $f = 0$ , an ever-steeper charge as $f$ approaches 1.

On the left, slide the prediction and compare the two curves; tick Compare squared error to see how timid squared error is. On the right is the cost over the tumour data as a single weight $w$ changes (with $b$ held fixed at −6). The log-loss cost is one smooth bowl. The squared-error cost is flat on the left, so a ball placed there gets no push at all.

One formula for both cases

Because $y$ is always 0 or 1, the two cases fit in one expression:

L\big(f, y\big) = -\,y \log(f) \;-\; (1 - y) \log(1 - f)

When $y = 1$ the second term vanishes and we’re left with $-\log f$ ; when $y = 0$ the first term vanishes and we’re left with $-\log(1 - f)$ . Averaging over the training set gives the logistic cost, also called log loss, logistic loss or binary cross-entropy:

J(\vec{w}, b) = -\frac{1}{m} \sum_{i=1}^{m} \Big[ y^{(i)} \log f_{\vec{w},b}(\vec{x}^{(i)}) + (1 - y^{(i)}) \log\big(1 - f_{\vec{w},b}(\vec{x}^{(i)})\big) \Big]

For logistic regression this cost is convex: one bowl, one bottom, no plateaus where the model is wrong. Gradient descent from any starting point heads to the best parameters.

Where it comes from: maximum likelihood

The log loss isn’t an arbitrary invention. In Math for ML, section 4 you met likelihood: the probability a model assigns to the data you actually observed, and the idea that a good model makes the observed data likely. Our model gives example $i$ probability $f$ if $y^{(i)} = 1$ and $1 - f$ if $y^{(i)} = 0$ . The likelihood of the whole training set is the product of these. Products of many small numbers are awkward, so take the logarithm, which turns the product into a sum, and flip the sign so that smaller is better. What you get is exactly $m \times J$ . Minimising the log loss is maximising the likelihood.

Computing it in NumPy

Here are the per-example losses for a few predictions. Notice how fast the confident mistakes grow:

import numpy as np

np.set_printoptions(suppress=True)
f = np.array([0.99, 0.9, 0.6, 0.5, 0.1, 0.001])
print("prediction f:     ", f)
print("loss if y = 1:    ", (-np.log(f)).round(3))
print("loss if y = 0:    ", (-np.log(1 - f)).round(3))
print("squared, if y = 1:", (0.5 * (f - 1) ** 2).round(3))

Exercise · Python

Compute the logistic cost

Complete compute_cost(X, y, w, b) so it returns the logistic cost $J = -\frac{1}{m} \sum \big[ y \log f + (1 - y) \log(1 - f) \big]$ with $f = g(X\vec{w} + b)$ , as a single number. Use np.log and vectorised NumPy (no loop needed).

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def compute_cost(X, y, w, b):
    m = len(y)
    f = sigmoid(X @ w + b)
    # average of the per-example log loss
    cost = None  # replace this line
    return cost

X = np.array([[0.5, 1.5], [1.0, 1.0], [1.5, 0.5], [3.0, 0.5], [2.0, 2.0], [1.0, 2.5]])
y = np.array([0, 0, 0, 1, 1, 1])
print(compute_cost(X, y, np.array([1.0, 1.0]), -3.0))   # about 0.3669