Decision boundaries

Where does logistic regression switch from "no" to "yes"? Exactly where w·x + b = 0. See that boundary as a line on a 2-D plane, and bend it into a circle with polynomial features.

Lesson · 14 min

In this lesson you will

Show that the 0.5 threshold is the same as the condition w·x + b ≥ 0
Draw the decision boundary of a two-feature model as a straight line
Use polynomial features to get curved boundaries such as a circle
Turn predicted probabilities into class predictions in NumPy

Logistic regression gives a probability. To make a decision we threshold it: predict $\hat{y} = 1$ when $f_{\vec{w},b}(\vec{x}) \ge 0.5$ . In this lesson we ask a geometric question: which inputs land on each side? The dividing line between them is the decision boundary, and seeing it is the best way to understand what a classifier can and can’t do.

The threshold in terms of z

Recall that $f = g(z)$ with $z = \vec{w} \cdot \vec{x} + b$ , and that the sigmoid crosses 0.5 exactly at $z = 0$ . Since $g$ only ever rises, it follows that

f_{\vec{w},b}(\vec{x}) \ge 0.5 \iff g(z) \ge 0.5 \iff z \ge 0 \iff \vec{w} \cdot \vec{x} + b \ge 0.

So the model predicts 1 whenever the linear part is zero or positive, and 0 when it’s negative. The sigmoid doesn’t move the boundary at all; it only turns distance from the boundary into confidence. The decision boundary is the set of points where

\vec{w} \cdot \vec{x} + b = 0.

With one feature, that’s a single point, $x = -b/w$ (the 4 cm from the last lesson). With two features it’s a line.

Two features, one line

A university admits students using two exam scores, $x_1$ and $x_2$ , both out of 10. Suppose the model is $z = w_1 x_1 + w_2 x_2 + b$ with $w_1 = 1$ , $w_2 = 1$ and $b = -10$ . The boundary is

x_1 + x_2 - 10 = 0 \quad\text{or}\quad x_1 + x_2 = 10,

a straight diagonal line. A student with scores (7, 6) has $z = 3 > 0$ , so they’re predicted “admitted”; one with (3, 4) has $z = -3$ , so “not admitted”. The further a point is from the line, the larger $|z|$ and the more confident the prediction: (9, 9) gives $z = 8$ and $g(8) \approx 0.9997$ .

The shading shows the probability everywhere on the plane: deep on one side, deep on the other, and pale along the boundary where the model is unsure. Drag $w_1$ , $w_2$ and $b$ :

Changing $b$ shifts the line without turning it.
Changing the ratio of $w_1$ to $w_2$ turns it. With $w_2 = 0$ , only exam 1 matters and the line is vertical.
Scaling all three by the same factor keeps the line in the same place but makes the shading sharper: same boundary, more confidence.

Raise the Threshold and the line moves towards the “admitted” corner: the model now needs more evidence before it says yes. Thresholds other than 0.5 matter when one kind of mistake costs more than the other, which Section 5 explores.

Curved boundaries from polynomial features

Now switch the dataset to Ring. Class 1 sits inside a circle and class 0 surrounds it. No straight line can separate them; the best one gets only about two thirds right.

In Section 2 you turned linear regression into a curve by adding polynomial features such as $x^2$ . The same trick works here. Give the model the extra features $x_1^2$ , $x_1 x_2$ and $x_2^2$ :

z = w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_1 x_2 + w_5 x_2^2 + b

The boundary $z = 0$ can now be a circle or an ellipse. For example, with $w_3 = w_5 = 1$ , everything else 0 and $b = -1$ , the boundary is $x_1^2 + x_2^2 = 1$ , the unit circle. The model is still linear in its parameters, so everything you know about training it still applies; it just looks at richer features. Set Polynomial degree to 2 and press Run to let gradient descent find the circle (the next lessons explain exactly how it does that).

Higher degrees allow wigglier boundaries. That power comes with a danger, overfitting, which has its own lesson later in this section.

Predictions in NumPy

In code you rarely draw the boundary. You compute probabilities for many examples at once, then compare with the threshold. With the examples stacked as the rows of a matrix X, X @ w gives every $\vec{w} \cdot \vec{x}^{(i)}$ in one go (Python for Data, section 1):

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

X = np.array([[7.0, 6.0], [3.0, 4.0], [9.0, 9.0], [5.0, 5.0]])   # exam 1, exam 2
w = np.array([1.0, 1.0])
b = -10.0

z = X @ w + b
p = sigmoid(z)
print("z:          ", z)
print("probability:", p.round(4))
print("prediction: ", (p >= 0.5).astype(int))

Notice the student at exactly (5, 5): $z = 0$ , probability 0.5, right on the boundary. By our convention ( $\ge$ ) they’re predicted 1.

Exercise · Python

From probabilities to predictions

Complete predict(X, w, b, threshold=0.5). X has one example per row, w is the weight vector and b the bias. Compute the probability $f = g(X\vec{w} + b)$ for every row, then return an array of integers: 1 where $f \ge$ threshold, 0 elsewhere. sigmoid is provided.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def predict(X, w, b, threshold=0.5):
    # 1. probabilities for every row of X
    # 2. compare with the threshold, return 0/1 integers
    return None  # replace this line

X = np.array([[7.0, 6.0], [3.0, 4.0], [5.0, 5.0]])
print(predict(X, np.array([1.0, 1.0]), -10.0))   # expect [1 0 1]