Gradient descent for logistic regression

We have a model and a convex cost. Training is now the same story as Section 1: start somewhere, compute the slope of the cost, take a step downhill, repeat. All we need is the gradient of the log loss.

The update rule

Gradient descent updates every parameter at the same time:

with learning rate $\alpha$, exactly as before. For the logistic cost the partial derivatives turn out to be

If that looks familiar, it should: it’s character for character the same as the gradient of linear regression’s squared-error cost. Each example contributes its error, prediction minus label, times its feature value. The difference is hidden inside $f$: here $f = g(\vec{w} \cdot \vec{x} + b)$, a probability, rather than $\vec{w} \cdot \vec{x} + b$ itself. Same code shape, different model.

Why the gradient is so tidy

It’s worth seeing why, because it’s a lovely use of the chain rule from Math for ML, section 3. Take one example and write the loss as a chain: the parameters give $z$, $z$ gives $f = g(z)$, and $f$ gives the loss $L = -y \log f - (1-y)\log(1-f)$. Then

Take the three links one at a time:

• Loss to prediction. Differentiating the logs gives $\dfrac{\partial L}{\partial f} = -\dfrac{y}{f} + \dfrac{1-y}{1-f} = \dfrac{f - y}{f(1-f)}$.
• Sigmoid. The sigmoid has a famously neat derivative: $\dfrac{dg}{dz} = g(z)\big(1 - g(z)\big) = f(1-f)$.
• Linear part. $z = \vec{w} \cdot \vec{x} + b$, so $\dfrac{\partial z}{\partial w_j} = x_j$ and $\dfrac{\partial z}{\partial b} = 1$.

Multiply them and the awkward $f(1-f)$ cancels perfectly:

Average over the $m$ examples and you have the formula above. This cancellation is no accident: the log loss and the sigmoid are a matched pair. With squared error the $f(1-f)$ factor would survive, and since it’s almost zero whenever the sigmoid is flat, that’s exactly the plateau problem from the last lesson.

Watch it train

Below, the model starts at $\vec{w} = 0$, $b = 0$: probability 0.5 everywhere, no boundary at all. Press Step a few times, then Run. The right panel plots the cost after every iteration.

Within a few dozen iterations the boundary swings into place and the cost falls quickly; after that, gradient descent spends hundreds of iterations making small improvements, mostly growing the weights so the shading gets more confident. On these 36 students it settles near cost 0.17 with 34 of 36 correct. Two students sit on the wrong side of any straight line, so no amount of training fixes them.

A few practical notes carry straight over from earlier sections:

• Learning rate. Too small and training crawls; too large and the steps overshoot. Try the $\alpha$ slider.
• Feature scaling (Section 2) matters just as much here. The widget quietly rescales both exam scores to a similar small range before training, which is why one $\alpha$ works well for every degree.
• Check convergence with the cost curve: it should fall every iteration and level off.

Vectorised in NumPy

With the examples as rows of X (shape $m \times n$), the whole gradient is three lines. The errors f - y form a vector of length $m$; multiplying by X.T sums error times feature for every feature at once.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x = np.array([1.0, 1.6, 2.1, 2.7, 3.2, 3.6, 4.4, 5.0, 5.5, 6.1, 6.8])
y = np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
X = ((x - x.mean()) / x.std()).reshape(-1, 1)      # one scaled feature, shape (11, 1)

w, b, alpha = np.zeros(1), 0.0, 1.0
for it in range(1001):
    f = sigmoid(X @ w + b)
    dj_dw = X.T @ (f - y) / len(y)
    dj_db = np.mean(f - y)
    w, b = w - alpha * dj_dw, b - alpha * dj_db
    if it % 250 == 0:
        cost = -np.mean(y * np.log(f) + (1 - y) * np.log(1 - f))
        print(f"iteration {it:4d}: cost {cost:.4f}  w {w[0]:.3f}  b {b:.3f}")

boundary_cm = x.mean() - b / w[0] * x.std()       # where w*x_scaled + b = 0, in cm
print(f"boundary at {boundary_cm:.2f} cm")

The boundary lands in the gap between the largest benign tumour (3.6 cm) and the smallest malignant one (4.4 cm). Because this data is perfectly separable, the cost would keep creeping towards zero forever as $w$ grows; in practice we stop when it stops improving meaningfully, or use regularisation, which you’ll meet shortly.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def compute_gradient(X, y, w, b):
    m = len(y)
    f = sigmoid(X @ w + b)
    err = f - y
    dj_dw = None  # replace: vector, one entry per feature
    dj_db = None  # replace: a single number
    return dj_dw, dj_db

X = np.array([[0.5, 1.5], [1.0, 1.0], [1.5, 0.5], [3.0, 0.5], [2.0, 2.0], [1.0, 2.5]])
y = np.array([0, 0, 0, 1, 1, 1])
print(compute_gradient(X, y, np.array([0.5, -0.5]), 0.3))

Check yourself