The sigmoid and logistic regression

The last lesson ended with a wish list: an output that stays between 0 and 1, flattens out far from the boundary and still has a simple boundary to learn. One small function delivers all of it.

The sigmoid function

The sigmoid (also called the logistic function) is

where $e \approx 2.718$ is the base of natural logarithms you met in Math for ML, section 1. It takes any real number $z$ and squashes it into the range between 0 and 1. Walk through a few values:

• Large positive $z$, say $z = 10$: $e^{-10}$ is tiny (about 0.000045), so $g(z) \approx 1/1 = 1$.
• Large negative $z$, say $z = -10$: $e^{10}$ is huge (about 22,000), so $g(z) \approx 1/22{,}000 \approx 0$.
• $z = 0$: $e^{0} = 1$, so $g(0) = 1/2$. The midpoint.

So the graph is an S: flat near 0 on the far left, rising steeply through 0.5 at $z = 0$, flattening near 1 on the far right. It never actually reaches 0 or 1. It is also symmetric: $g(-z) = 1 - g(z)$.

NumPy’s np.exp works on whole arrays, so the sigmoid is one line:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-8, 8, 161)
print("g(-10), g(0), g(10) =", sigmoid(np.array([-10.0, 0.0, 10.0])).round(5))
bpl.plot(z, sigmoid(z), kind="line", title="The sigmoid g(z)")

Logistic regression: a line, then a squash

Logistic regression feeds a linear model into the sigmoid. First compute the same quantity as linear regression,

using the vector notation from Section 2 (with one feature it’s just $z = wx + b$). Then squash it:

The output is always between 0 and 1, and we read it as a probability:

“The probability that $y$ is 1, given the features $\vec{x}$, for these parameters.” If the model says 0.83 for a tumour, it estimates an 83 % chance it’s malignant, and therefore a 17 % chance it’s benign. You met conditional probability in Math for ML, section 4; this is that idea put to work.

To turn the probability into a yes-or-no answer, apply a threshold, usually 0.5: predict $\hat{y} = 1$ when $f \ge 0.5$ and $\hat{y} = 0$ otherwise.

Explore the S-curve

Here are the same tumours, now with a sigmoid model. Move $w$ and $b$ and watch the curve. Switch on Show g(z) panel to see the two stages separately: the linear part turns each tumour size into a $z$, and the sigmoid turns $z$ into a probability.

Three things to notice:

• $w$ sets the steepness. A large $w$ makes the S sharp, so the model jumps from “surely benign” to “surely malignant” over a few millimetres. A small $w$ makes it gentle and hesitant. A negative $w$ flips it, so larger tumours would look less likely to be malignant.
• $b$ slides the curve sideways. The curve crosses 0.5 where $z = 0$, that is where $wx + b = 0$, at $x = -b/w$. With $w = 2$ and $b = -8$ that’s $x = 4$ cm.
• Far points no longer matter much. Tick Add very large tumours. The sigmoid is already almost exactly 1 out there, so these points are predicted correctly and the boundary stays put.

Your turn: a sigmoid that works on arrays

Machine learning code calls the sigmoid on whole arrays of $z$ values at once, one per example.

import numpy as np

def sigmoid(z):
    # Return 1 / (1 + e^(-z)), element by element.
    return None  # replace this line

print(sigmoid(0))                         # should be 0.5
print(sigmoid(np.array([-2.0, 0.0, 2.0])))

Check yourself