Accuracy, Precision, Recall

Imagine a doctor announces a new test for a rare disease that is "99% accurate." Sounds fantastic — until you learn the disease affects only 1 in 100 people. A "test" that never bothers looking and just prints "healthy" for everyone would also score 99% accuracy, while catching precisely nobody who is actually sick. Accuracy alone told us nothing useful.

This is the trap of an imbalanced problem — one where the classes aren't roughly equal in size. To see past it, we need sharper tools built from the confusion matrix: for every prediction, was it a

true positive (TP) — predicted positive, and really was;
false positive (FP) — predicted positive, but actually wasn't;
false negative (FN) — predicted negative, but actually was positive; or
true negative (TN) — predicted negative, and really wasn't?

Precision and recall, side by side

From those four counts come two metrics that ask two different questions:

Precision = \dfrac{TP}{TP + FP} — of everything the model flagged positive, how much really was positive? High precision means few false alarms.
Recall = \dfrac{TP}{TP + FN} — of everything that really was positive, how much did the model actually catch? High recall means few missed cases.

Let's make this concrete with a spam filter that checks 100 emails, of which 20 really are spam and 80 are not. Suppose the filter flags 25 emails as spam, and of those, 18 really are spam (so 7 genuine emails get wrongly caught) — and it lets 2 real spam emails slip through to the inbox:

	Actually spam	Actually not spam
Predicted spam	TP = 18	FP = 7
Predicted not spam	FN = 2	TN = 73

Here \text{precision} = \tfrac{18}{18+7} = \tfrac{18}{25} = 0.72 (72% of what the filter flagged really was spam) while \text{recall} = \tfrac{18}{18+2} = \tfrac{18}{20} = 0.9 (it caught 90% of all the real spam). Same confusion matrix, two genuinely different numbers — because they're answering two genuinely different questions.

Worked example: the useless 99%-accurate test

Back to the disease that affects 1 in 100 people. Out of 1,000 patients, 10 truly have the disease. The "always say healthy" test produces this confusion matrix:

	Actually sick	Actually healthy
Predicted sick	TP = 0	FP = 0
Predicted healthy	FN = 10	TN = 990

Accuracy = \tfrac{0 + 990}{1000} = 0.99 — a glowing 99%. But recall = \tfrac{0}{0+10} = 0: it catches none of the sick patients, ever. (Precision isn't even well-defined here — the test never predicts "sick" at all, so there's nothing to divide by.) A real, useful test — even one that makes some mistakes, say TP = 8, FP = 5, FN = 2, TN = 985 — has a slightly lower accuracy of \tfrac{8+985}{1000} = 0.993 but an actually meaningful recall of \tfrac{8}{8+2} = 0.8. Accuracy alone would have told you to prefer the useless test. Precision and recall tell you the truth.

Move the threshold

A classifier usually doesn't just output "positive" or "negative" — it outputs a score between 0 and 1, and we pick a threshold above which we call it "positive." Each example below has a true label and a model score; slide the decision threshold and watch the confusion-matrix counts and all three metrics update live. At the default threshold of 0.5, this toy dataset gives TP = 5, FP = 3, FN = 2, TN = 4 — precision = \tfrac{5}{8} \approx 0.63, recall = \tfrac{5}{7} \approx 0.71. Raise the threshold and the model becomes fussier: it flags fewer positives, so precision tends to rise while recall falls. Lower it and the model becomes trigger-happy: recall rises while precision falls. There's no free lunch — you are always trading one against the other.

Which mistake would you rather make?

Precision and recall matter in different amounts depending on which kind of mistake is more expensive:

For a spam filter, a false positive means an important email — a job offer, a message from a friend — vanishes into the spam folder. That's the more painful mistake, so you favour precision: better to let a little spam through than to bury real mail.
For cancer screening, a false negative means a real case is missed and sent home undiagnosed. That's far more dangerous than a false alarm that leads to a follow-up test, so you favour recall: better to over-flag a few healthy people than miss someone who is sick.

When you need one single number that balances both, use the F1 score, the harmonic mean of precision and recall, F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} For the spam filter above, F_1 = \tfrac{2 \times 0.72 \times 0.9}{0.72 + 0.9} \approx 0.8. The harmonic mean punishes a metric that's very low far more than a plain average would — a model with precision 1.0 but recall 0.01 gets an F1 near 0.02, not 0.5, because being nearly useless in one respect should drag the combined score down hard. Never trust accuracy alone on imbalanced data; with evaluation in hand, we're ready for the most powerful models of all: neural networks.

Whenever one class vastly outnumbers the other, accuracy becomes almost meaningless as a headline number. If 99% of patients are healthy, a model can hit 99% accuracy by memorising the single word "healthy" and never looking at the data at all — its recall for the disease is a flat zero. The same trap appears everywhere: fraud detection (almost all transactions are legitimate), rare-event alerts, even spam filters if spam is much rarer than real mail. The fix is always the same — check precision and recall (or F1) for the class you actually care about, not just overall accuracy, especially whenever one class is much rarer than the other.

Long before anyone wrote "confusion matrix," Aesop told the story of a shepherd boy who raised a false alarm about a wolf again and again for fun. His "wolf detector" had perfect recall in one sense — he never once failed to shout when there really was danger — but his precision was terrible, because almost every alarm was a false positive. By the time a real wolf showed up, the villagers had learned to ignore him, and it cost the flock dearly. The moral is a precision/recall lesson in disguise: a detector that cries "positive!" constantly will eventually catch every real case, but if nobody trusts its alarms any more, high recall stops being worth anything.

This isn't just a classroom exercise — production spam filters, fraud-detection systems, and medical-screening tools all explicitly choose where to sit on the precision/recall trade-off by setting their decision threshold. A hospital screening tool for a dangerous disease is deliberately tuned toward high recall, accepting more false alarms (which just mean a follow-up test) in exchange for almost never missing a real case. An email provider tunes the opposite way, accepting that a little spam gets through so that your important mail practically never vanishes. Neither threshold is "more correct" than the other — they're different answers to the question "which mistake costs us more here?"