More than a fork in the road

Deciding "spam or not spam" is a simple fork in the road — two doors, pick one. But most real classification problems aren't that tidy. Which of the 10 digits 0–9 did someone just write by hand? Which of 1000 dog breeds is in this photo? Which of thousands of possible commands did you just say to a voice assistant? These need a classifier that can pick one winner out of many options at once — that's multiclass classification.

Two clean strategies extend our two-class tools to many classes:

• One-vs-rest. Train one binary classifier per class ("class 3 vs everything else"). To predict, run them all and pick the one most confident it's a match.
• Softmax. Generalize the sigmoid to output a whole probability distribution over the classes at once — one number per class, all positive and summing to 1.

Softmax, informally: spend 100% of your confidence

Instead of computing one lonely sigmoid probability, imagine the classifier first computes a raw score for every possible class — a number that's bigger when the model thinks that class is more likely, with no particular scale or upper limit. Softmax's job is to take that messy pile of scores and turn it into a tidy set of probabilities that add up to exactly 1 — as if the model has exactly "100% of confidence" to spend, and has to divide it across every option.

Formally, softmax takes the raw scores z_1, \dots, z_C (one per class, C classes total) and produces:

The exponential e^{z_c} makes every score positive (even a negative raw score turns into a small positive number), and dividing by the sum of all the exponentials forces the results to add up to one. The bigger a class's raw score relative to the others, the bigger its share of that "100%."

Worked example: cat, dog, or bird?

Say a network looks at a photo and produces three raw scores — one per class — before softmax gets involved: z_{\text{cat}} = 2.0, z_{\text{dog}} = 1.0, z_{\text{bird}} = 0.1. Cat has the highest raw score, so we'd expect it to come out on top — but by how much? Run softmax step by step:

• Exponentiate each score: e^{2.0} \approx 7.389, e^{1.0} \approx 2.718, e^{0.1} \approx 1.105.
• Add them up: 7.389 + 2.718 + 1.105 = 11.212.
• Divide each by that total: cat \approx 7.389/11.212 \approx 0.659, dog \approx 2.718/11.212 \approx 0.242, bird \approx 1.105/11.212 \approx 0.099.

Check it: 0.659 + 0.242 + 0.099 = 1.000 — exactly one, as promised. The classifier reports "66% cat, 24% dog, 10% bird" and predicts cat, the highest of the three. Try it yourself:

Now compare a closer contest: scores z_{\text{cat}} = 1.0, z_{\text{dog}} = 0.9, z_{\text{bird}} = 0.2. Feed those into the same code and you'll get roughly 42% cat, 38% dog, 19% bird — still a "cat" prediction, but a nail-biting one. The raw scores were close together, and softmax faithfully reflects that closeness by keeping the probabilities close together too, instead of pretending the decision was obvious.

The alternative: one-vs-rest

Softmax scores every class together, in one shot. One-vs-rest instead builds C separate, independent binary classifiers, each asking one narrow question: "is it THIS class, or not?" On the same cat/dog/bird photo, that means training three separate yes/no classifiers — "cat vs. not-cat," "dog vs. not-dog," "bird vs. not-bird" — and asking all three:

• "Is it a cat?" → 0.70 confident yes.
• "Is it a dog?" → 0.55 confident yes.
• "Is it a bird?" → 0.20 confident yes.

To predict, just take whichever classifier shouted loudest: cat, at 0.70. Notice something odd, though — those three numbers add up to 1.45, not 1. That's completely fine for one-vs-rest, because each of the three classifiers was trained separately and has no idea the other two even exist. Softmax's numbers are forced to agree with each other and sum to exactly one; one-vs-rest's numbers are three independent opinions that happen to be compared side by side at the end.

Which one should you use?

Both strategies solve the same problem, but they trade off differently:

• One-vs-rest is conceptually simple — it's just ordinary binary classification, repeated C times. That makes it easy to reuse existing binary tools and easy to add a brand-new class later without retraining everything from scratch. Its downside is exactly the "photo finish" problem: because the classifiers are trained independently, nothing stops two of them from both being 90\% confident, or all three being only 20\% confident, leaving genuinely ambiguous cases.
• Softmax trains all the classes together as one system, so its probabilities are automatically consistent with each other and always sum to one — there's never a "which classifier do I trust more" ambiguity. The trade-off is that softmax needs to see all the classes at once during training; adding one new class later usually means retraining the whole output layer, not just bolting on one more piece.

In modern deep learning, softmax's clean, jointly-trained probabilities usually win out, which is why it's the default final layer for image classifiers, language models choosing the next word, and almost anything else built from neural networks today.

A geometric picture: carving up the space

Whichever method you use, the end result is the same kind of map: the space of possible inputs gets carved into regions, one per class. Below, each class has a representative point (a centroid), and a query is assigned to whichever class it's closest to. Drag the query across a border and watch its predicted class flip — a simple stand-in for the decision boundaries that softmax and one-vs-rest both draw, just with a distance rule instead of scores or probabilities.

Softmax and the standard classifier

Paired with cross-entropy loss, softmax is the standard final layer of almost every classification neural network — the last stop before the model commits to an answer. One-vs-rest is simpler to reason about and still used with classifiers that don't naturally output multiple scores at once (like some logistic regression setups), but softmax's all-at-once, properly-normalised probabilities are why it dominates modern deep learning.

The pairing works like this: once softmax has produced a probability for every class, training just needs to know one number — the probability the model assigned to whichever class was actually correct. Cross-entropy loss for a multiclass example is L = -\log(p_{\text{correct class}}) — take that one probability, negate its log, and you have exactly the same "small loss when confident and right, huge loss when confident and wrong" behaviour as the two-class version, just applied across as many classes as you like.