The Decision Boundary

Every classifier, once it's trained, is secretly drawing something you can picture: an invisible line (or, in more than two dimensions, a curved or flat surface) cutting through feature space. Everything on one side of that line gets one label; everything on the other side gets the other. That line is the decision boundary, and it is often the single clearest way to see what a trained classifier actually learned — far clearer than squinting at a list of numbers.

With two features, the boundary of a linear classifier is a single straight line splitting the plane in two. It is exactly the set of points where the model is perfectly undecided, scoring 0.5:

\vec{w}\cdot\vec{x} + b = 0.

On one side \vec{w}\cdot\vec{x} + b > 0 and the model predicts class 1; on the other it's negative and predicts class 0. The weight vector \vec{w} points perpendicular to the boundary, toward the positive class — a direct echo of the dot product's sign.

Where the boundary comes from

This isn't a separate idea bolted onto logistic regression — it falls straight out of it. Logistic regression outputs a probability p = \sigma(z) where z = \vec{w}\cdot\vec{x} + b. The sigmoid crosses its own midpoint — the exact fifty-fifty point, \sigma(0) = 0.5 — precisely when its input z is zero. So the boundary \vec{w}\cdot\vec{x} + b = 0 isn't a new formula to memorise; it is simply "wherever z = 0," which is wherever the predicted probability sits at exactly 0.5.

Steer the boundary with the weights

The two weights and the bias are the boundary. Adjust them and watch the dividing line swing and shift, the shaded half-planes following along. The arrow is the weight vector \vec{w}, always at right angles to the boundary and pointing into the class-1 region.

Try dragging w_1 and w_2 toward zero while keeping b fixed — the boundary flattens and the arrow shrinks, because the classifier is losing its ability to tell the classes apart at all. Now drag b alone: the line slides sideways without rotating, because the bias only shifts where the boundary sits, never its tilt.

Three features: the boundary becomes a plane

Add a third feature and the dividing line grows into a flat plane slicing space into two half-spaces — points on one side get one label, points on the other get the other. Here are two classes of points and the plane x + y + z = 0 that separates them; drag to rotate and watch the two coloured clouds split cleanly by the sheet.

Worked example: find the boundary, then use it

Suppose a trained 2-feature linear classifier has weights w_1 = 2, w_2 = -1, and bias b = -2. Its boundary is:

2x_1 - x_2 - 2 = 0 \quad\Longleftrightarrow\quad x_2 = 2x_1 - 2.

That's a line of slope 2 crossing the x_2-axis at -2. To find out which side is which, test one known point — the origin (0,0) is the easiest:

2(0) - (0) - 2 = -2 < 0 \;\Rightarrow\; \text{class 0}.

So the origin's side of the line is the class-0 region, and the opposite side is class 1. Now classify a brand-new point, (1, -1), that the model has never seen before — just plug it into the same formula:

2(1) - (-1) - 2 = 2 + 1 - 2 = 1 > 0 \;\Rightarrow\; \text{class 1}.

No probabilities, no sigmoid needed at prediction time — once you have the boundary, classifying a new point is just "which side of the line is it on?" Compare two more new points: (4, 0) gives 2(4) - 0 - 2 = 6, a large positive number, so the model predicts class 1 confidently. But (1, 0) gives 2(1) - 0 - 2 = 0 — sitting exactly on the boundary, the model's most uncertain possible call. The size of \vec{w}\cdot\vec{x}+b, not just its sign, tells you how far a point sits from the boundary and so how confident the prediction is.

Not just any separating line will do

Logistic regression is happy with any line that has the training points sorted correctly on either side — even one that grazes right past a few points with barely any room to spare. That can be a problem: a boundary squeezed tight against the data is more likely to misclassify a new point that lands just slightly differently than the training examples did. Some classifiers, like support vector machines, take this a step further and deliberately search for the boundary with the widest possible cushion — the margin — between itself and the nearest point of each class, on the theory that a boundary with breathing room generalises better to data it hasn't seen yet.

When a straight line isn't enough

A linear boundary is a straight line (a flat hyperplane in higher dimensions) — simple, fast, and often enough. But imagine a dataset shaped like a target: class A packed in a small disc near the origin, and class B scattered in a ring all the way around it. No straight line, however you tilt or shift it, can ever put every class-A point on one side and every class-B point on the other — the inner disc is surrounded on all sides by the ring.

When classes interlock like this, you reach for curved boundaries: add polynomial features (so a straight boundary in the new, richer feature space becomes a curve back in the original one), or let a neural network bend the boundary into whatever shape the data demands. But every one of them is still answering the same question, "which side are you on?" — just with a fancier dividing surface.

Two traps that catch people reading a decision boundary:

Being close to the boundary means being unsure. A point that lands just a hair on the class-1 side is barely different from landing just a hair on the class-0 side — the model is nearly a coin flip there. A prediction far from the boundary, by contrast, is one the model is confident about. When you're deciding how much to trust a prediction, distance from the boundary matters as much as which side it's on.
No amount of retuning rescues a straight line from a curved problem. For data arranged like two concentric rings, every possible choice of w_1, w_2, b still only draws a straight line — and a straight line can never encircle the inner ring while excluding the outer one. If your linear classifier is stuck at poor accuracy no matter how you train it, the shape of the boundary it's capable of drawing, not the training process, may be the real limit.

Because it works. Plotting two features on a chart, colouring each training point by its true class, and drawing the model's boundary through them like a coastline separating two coloured "countries" is one of the most common — and most useful — diagrams in all of machine learning teaching. It turns an abstract list of weights into something you can look at and immediately judge: does this shape make sense for this data?

That same picture also explains a famously strange phenomenon called an adversarial example: an image an AI classifies as, say, a "panda" can sometimes be nudged by a tiny, human-imperceptible change so it crosses to the wrong side of the model's boundary and gets classified as something absurd, like a "gibbon" — with the model suddenly very confident about it. Knowing that a boundary is just a surface in feature space, and that some points sit dangerously close to it, makes that trick far less mysterious.