Cross-Entropy Loss

How do you grade a guess?

A spam filter doesn't just say "spam" or "not spam" — underneath, it outputs a probability, like "97% sure this is spam." To train that filter, we need a way to score how good each probability guess was, once we find out the real answer. Squared error — the tool we use for plain numbers — turns out to be the wrong ruler for probabilities: it doesn't punish a wildly overconfident wrong answer nearly hard enough.

What we actually want is a loss that is gentle when the model is confidently right, and brutal when it's confidently wrong — so the safest strategy is always to be honest about your uncertainty. That loss is called cross-entropy (also known as log loss). For a single example with true label y \in \{0, 1\} and predicted probability p (the model's estimate that y = 1):

L = -\big[\,y\log p + (1 - y)\log(1 - p)\,\big].

It looks like two terms added together, but really it's a switch: only one term is ever "on" for a given example, and which one depends on the true label. Think of it like a strict exam marker: the marker doesn't care how eloquently you argued your answer — only how much probability you were willing to stake on the option that turned out to be correct. Stake it all on the right answer and you pay almost nothing; stake it all on the wrong one and the bill is severe.

Unpacking the switch, term by term

Look at what happens to each half of the formula depending on y:

If the truth is y = 1, then (1-y) = 0, so the whole second term (1-y)\log(1-p) vanishes — it's multiplied by zero. The loss collapses to just L = -\log p.
If the truth is y = 0, then it's the first term that's multiplied by y = 0 and disappears. The loss collapses to L = -\log(1-p) — note 1-p is the model's probability of the class actually being 0.

Either way, the formula boils down to one simple English sentence: "take the negative log of the probability the model assigned to the class that actually happened." Since probabilities live between 0 and 1, and \log of a number in that range is negative, the minus sign flips the loss positive — so a loss of 0 means a perfect, fully-confident correct guess, and bigger numbers mean worse guesses.

Three worked examples

The best way to feel the shape of this formula is to plug in real numbers. Suppose the true label is always y = 1 (the email really is spam), and look at three different guesses the model might have made:

Correct and confident: the model says p = 0.95. The loss is -\log(0.95) \approx 0.051 — tiny. Being right and sure of it is almost free.
Wrong and confident: the model says p = 0.05 (it was almost sure this was not spam). The loss is -\log(0.05) \approx 2.996 — nearly sixty times bigger than the first example, for a guess that was only 0.90 further from the truth. That's the harsh penalty in action: confidently wrong is punished far out of proportion to a small numerical miss.
Uncertain — "played it safe": the model shrugs and says p = 0.5. The loss is -\log(0.5) \approx 0.693 — a moderate, middle-of-the-road penalty. Notice this number doesn't depend on whether the truth is y=1 or y=0: guessing "I have no idea" costs the same either way, and it costs less than being confidently wrong, but more than being confidently right.

The same pattern holds in mirror image when the true label is y = 0 (the email really is not spam): a model predicting p = 0.05 (correctly, and confidently, saying "not spam") pays only -\log(1 - 0.05) = -\log(0.95) \approx 0.051 — the same tiny loss as before, just triggered by the other term of the formula. Cross-entropy doesn't care which label is "1" and which is "0"; it only cares whether the probability assigned to whatever actually happened was high or low.

Run the numbers yourself:

function crossEntropyLoss(y: number, p: number): number { return -(y * Math.log(p) + (1 - y) * Math.log(1 - p)); } console.log("confident & correct (y=1, p=0.95):", crossEntropyLoss(1, 0.95).toFixed(3)); console.log("confident & WRONG (y=1, p=0.05):", crossEntropyLoss(1, 0.05).toFixed(3)); console.log("played it safe (y=1, p=0.50):", crossEntropyLoss(1, 0.50).toFixed(3));

The ordering — small, then moderate, then huge — is exactly the incentive we want to give a model during training: don't just try to be right, try to be calibrated, and never bluff.

The cost of being wrong, as a curve

Pick the true label and watch the loss as the predicted probability slides from 0 to 1. Toward the correct end it dips smoothly to zero; toward the wrong end it doesn't just rise, it rockets upward. That steepness near the wrong extreme is the whole point — it's what makes cross-entropy so much harsher on overconfident mistakes than a plain "how far off was the number" score would be.

Why the near-vertical cliff is on purpose

A model that says "100% spam" about a real, important email should be punished without mercy — cross-entropy's blow-up toward infinity does exactly that, discouraging the model from ever claiming total certainty unless it has earned it. Averaged over every example in the training set and minimised by gradient descent, this one formula trains logistic regression and the output layer of nearly every classification neural network — from spam filters to the model reading handwriting on a cheque.

Compare that to squared error, (y-p)^2, on the same confidently-wrong guess (y=1, p=0.05): squared error only gives (1-0.05)^2 = 0.9025 — a fairly ordinary number, nothing dramatic. Squared error treats "off by 0.95" the same whether it happened at the confident extreme or in the middle of the range. Cross-entropy, by contrast, saves its harshest punishment specifically for confident mistakes, which is exactly the behaviour we want to train into a classifier.

Averaging over a whole dataset

Everything so far has been the loss for one example. In practice a model is trained on a whole batch of examples at once, and what actually gets minimised is the average loss across every one of them. Suppose a tiny dataset of four emails has these true labels and predictions:

Email 1 — true label y=1 (spam), predicted p=0.90: loss \approx 0.105.
Email 2 — true label y=0 (not spam), predicted p=0.10: loss -\log(1-0.10) \approx 0.105.
Email 3 — true label y=1 (spam), predicted p=0.20 (a bad, overconfident-in-the-wrong-direction guess): loss -\log(0.20) \approx 1.609.
Email 4 — true label y=0 (not spam), predicted p=0.50 (uncertain): loss \approx 0.693.

The dataset's overall loss is the mean of those four numbers: (0.105 + 0.105 + 1.609 + 0.693) / 4 \approx 0.628. Notice how much Email 3 dragged the average up on its own — one badly overconfident mistake can outweigh several good guesses put together. That's exactly the signal gradient descent needs: it nudges every weight in the direction that would have shrunk that one painful loss the most, over and over, across the whole dataset, until the average loss is as small as it can be.

Beyond two classes

Everything here has been the binary version, for a yes/no question like "spam or not spam." Real classifiers often have to choose between many classes at once — which of 10 handwritten digits, which of 1000 animal species. That's the job of multiclass classification, which extends this same idea: instead of one probability p, the model outputs one probability per class (via softmax), and cross-entropy loss becomes -\log(p_{\text{correct class}}) — take the negative log of whichever probability the model assigned to the class that actually happened. The binary formula above is just the special case where there are only two classes to choose between.

Look again at -\log p as p \to 0: the loss heads to infinity. That's not just a big number — \log(0) is mathematically undefined. If a model is exactly 100% confident and turns out to be wrong, the formula asks for the negative log of zero, which blows up to negative infinity, and the loss becomes infinite. A single infinite loss would wreck the average over the whole dataset and stall training completely.

Real implementations dodge this quietly: they clip every predicted probability to stay just inside the open range, e.g. never letting p go below something like 10^{-7} or above 1 - 10^{-7}. The model can get astronomically close to "certain," but the loss function never actually divides by, or takes the log of, a true zero. It's an unglamorous engineering fix protecting an elegant piece of maths.

Yes — and it happens constantly. Accuracy only asks "did you pick the right class?" — a yes/no question. Cross-entropy asks "how confident, and how well-calibrated, was your probability?" — a much finer-grained question. A model can keep every single one of its predictions on the correct side of the decision boundary (accuracy frozen, unchanged) while quietly tightening its probabilities — turning a shaky "60% sure" into a firm "92% sure" on examples it was already getting right. Accuracy doesn't move a millimetre, but cross-entropy loss drops, because the model has become better calibrated, not just better at picking sides.

This is exactly why cross-entropy, not accuracy, is the number used to drive training: it keeps giving useful feedback long after accuracy has flatlined.

"Cross-entropy" isn't a made-up machine-learning word — it was borrowed wholesale from information theory, the field Claude Shannon founded in 1948 to figure out how efficiently messages could be squeezed down a telegraph wire. Shannon defined the "entropy" of a message as how many bits of surprise it carries on average: a message you could have predicted perfectly carries zero surprise (zero bits), while a wildly unexpected message carries a lot. Cross-entropy extends that idea to compare two probability distributions — the one you predicted versus the one that actually happened — and that is precisely what a classifier's loss needs to measure.

It's also worth asking: why not just train a model by directly counting how many answers it gets wrong (its error rate)? Because "count the mistakes" is a staircase function — flat, then a sudden jump, then flat again — with zero slope almost everywhere. Gradient descent needs a smooth hill to walk down, and cross-entropy, unlike a raw mistake-count, changes gradually as the predicted probabilities nudge up or down. That smoothness is the unglamorous reason cross-entropy — not accuracy — is the number actually staring back at the optimiser during training for nearly every classifier on Earth.