Entropy, Cross-Entropy and KL Divergence

Three quantities from information theory quietly run through the whole of machine learning: the loss a classifier minimises, the criterion a decision tree splits on, the objective a language model is trained against — all of them are the same handful of ideas wearing different clothes. This page introduces them together, because their real beauty is how they connect: entropy measures uncertainty, cross-entropy measures the cost of being wrong about it, and KL divergence is exactly the gap between the two.

Claude Shannon set the whole subject going in 1948 with one question: how many yes/no questions — bits — does it take, on average, to pin down the outcome of a random event?

Entropy: how much uncertainty is in a distribution

For a probability distribution p over outcomes, Shannon entropy is

H(p) = -\sum_i p_i \log_2 p_i.

Read it as the average number of bits you need to encode an outcome drawn from p, using the best possible code. A rare outcome (p_i small) carries a lot of "surprise," -\log_2 p_i bits; entropy is the average surprise, weighted by how often each outcome actually happens. A certain event has entropy 0 (no questions needed); a uniform event has the most entropy of all.

For a coin with probability p of heads there are only two outcomes, and the entropy — the binary entropy function — is H(p) = -p\log_2 p - (1-p)\log_2(1-p). Its shape tells the whole story:

It is 0 at the ends (a two-headed or two-tailed coin is perfectly predictable) and rises to a peak of exactly 1 bit at p = 0.5 — a fair coin is the most uncertain of all, needing one full yes/no question per flip. This same curve is the entropy a decision tree tries to reduce when it chooses a split: a pure node has entropy 0, a 50/50 node has entropy 1.

Cross-entropy: the cost of the wrong code

Now suppose the true distribution is p, but you build your code (your model's beliefs) assuming a different distribution q. The average number of bits you now spend is the cross-entropy:

H(p, q) = -\sum_i p_i \log_2 q_i.

Outcomes still occur with their true frequencies p_i, but you pay -\log_2 q_i bits for each because your code was designed for q. If your model is perfect (q = p) this collapses back to H(p). If your model is confidently wrong — it assigns a tiny q_i to an outcome that keeps happening — the cost blows up toward infinity. That penalty on confident mistakes is exactly why cross-entropy is the loss classifiers and neural networks minimise: driving cross-entropy down means making the model's predicted q match reality p.

KL divergence: the extra bits from being wrong

Cross-entropy is always at least as big as entropy — using the wrong code can never help. The excess, the bits you waste purely because q \neq p, is the Kullback–Leibler divergence:

The decomposition H(p,q) = H(p) + D_{\mathrm{KL}}(p\,\|\,q) ties the trio together in one line: the true uncertainty you can never remove, plus the avoidable waste from a wrong model. Because H(p) doesn't depend on your model, minimising cross-entropy is identical to minimising KL divergence — training a classifier is literally pushing its beliefs q as close to the truth p as the data allow.

A worked coin example

Let the truth be a biased coin, p = (0.25, 0.75) heads/tails. Its entropy is

H(p) = -0.25\log_2 0.25 - 0.75\log_2 0.75 \approx 0.81 \text{ bits}.

Suppose your model wrongly believes the coin is fair, q = (0.5, 0.5). The cross-entropy is

H(p,q) = -0.25\log_2 0.5 - 0.75\log_2 0.5 = 1 \text{ bit},

so the KL divergence — your wasted bits — is D_{\mathrm{KL}}(p\,\|\,q) = 1 - 0.81 = 0.19 bits. You spend a fifth of a bit per flip more than necessary, purely because your model guessed the wrong bias. Fix the model to q = p and that waste vanishes to exactly 0.

The \log_2 is what puts the answer in bits: one bit is the information in a single fair coin flip, because it takes exactly one yes/no question to resolve. Switch to the natural log \ln and the unit becomes nats; switch to \log_{10} and it's dits. Machine-learning libraries almost always use nats internally (natural log is smoother to differentiate), which is why a reported cross-entropy loss rarely looks like a tidy number of bits — same quantity, different ruler.

It is tempting to treat D_{\mathrm{KL}} as "how far apart" two distributions are, but it breaks the rules of a distance: it is asymmetric (D_{\mathrm{KL}}(p\,\|\,q) \neq D_{\mathrm{KL}}(q\,\|\,p)) and violates the triangle inequality. The asymmetry is meaningful, not a defect: D_{\mathrm{KL}}(p\,\|\,q) punishes a model q savagely for assigning near-zero probability to something the truth p says happens — but is relaxed about q wasting probability on things that never occur. Pick the wrong direction and your model chases the wrong failure. When you truly need a symmetric measure, use the Jensen–Shannon divergence, which is built from two KL terms.

Learn this on Kaggle

Kaggle Learn's Feature Engineering course uses information-theoretic thinking directly: mutual information (built from entropy) scores how much each feature tells you about the target. Working through it makes entropy and cross-entropy concrete on real tables, and it sets up the mutual-information feature scores you'll meet next.