Type I and Type II Errors

Imagine a shepherd boy watching a flock. Every hypothesis test he ever runs — "is that a wolf?" — can go wrong in two opposite ways. He can cry wolf when the bushes were only rustling in the wind (a false alarm), or he can stay silent while a real wolf slips in among the sheep (a missed threat). These are not the same mistake, they are not equally bad, and — crucially — you cannot make both impossible at once.

Statistics gives these two blunders names: the false alarm is a Type I error, the missed threat a Type II error. Grasping the see-saw between them is exactly what separates real statistical literacy from mechanically chanting "p < 0.05." A smoke detector, a courtroom, a cancer screen, and a drug trial are all, underneath, the same shepherd deciding how afraid of each mistake to be.

The two ways to be wrong

A verdict in a hypothesis test can be wrong in two different ways — and they are not the same mistake. Lay the truth against our decision:

Type I error — we reject a true H_0. A false positive: we cry "effect!" when there is none. Its long-run rate is exactly the significance level, \alpha.
Type II error — we fail to reject a false H_0. A false negative: a real effect slips past us. Its rate is written \beta.

The two correct verdicts are rejecting a false H_0 and failing to reject a true one. Notice the pairing: H_0 is the "nothing unusual" default (no wolf, no disease, no effect), and each error is naming which direction we got the default wrong.

Power: catching a real effect

If \beta is the chance of missing a real effect, then 1 - \beta is the chance of catching it. This is the power of the test:

\text{power} = 1 - \beta = \mathbb{P}(\text{reject } H_0 \mid H_1 \text{ true}).

A powerful test rarely lets a genuine effect through. So a good test wants both \alpha small (few false alarms) and power high (few misses). A test with power 0.8 will detect a real effect 80% of the time — and miss it the other 20%. Underpowered studies, with power far below that, are quietly one of the biggest problems in modern science.

Worked example 1 — a medical test

A screening test decides between H_0 = "patient is healthy" and H_1 = "patient has the disease." Spell out each error concretely:

Type I (false positive): telling a healthy person they are sick. The alarm went off with no fire — worry, more tests, maybe needless treatment.
Type II (false negative): telling a sick person they are fine. The real disease slipped past — the most dangerous kind of miss, because the patient goes home untreated.

Which is worse? For a serious, treatable illness, most people judge the Type II error (missing it) the graver one, and are willing to tolerate more false positives to catch every true case. That is a deliberate choice about where to sit on the see-saw — not a fact the maths hands you.

Worked example 2 — the smoke detector

Here H_0 = "no fire." A Type I error is the detector shrieking when you've merely burnt the toast; a Type II error is the detector staying silent while the kitchen is genuinely ablaze. One is annoying; the other can be fatal.

So we deliberately build smoke detectors to minimise Type II — never, ever miss a real fire — and we simply accept the cost: a jumpy detector that goes off at steam and toast. We would rather be annoyed a hundred times than incinerated once. Compare that with a spam filter, where a Type I error (a real email lost to the junk folder) may hurt more than a Type II (one spam message reaching your inbox), so the trade is tuned the other way. Which error is worse depends entirely on the stakes.

The two errors, side by side

The left bell is the sampling distribution under H_0; the right bell is the distribution under H_1 (a real effect shifts the mean across). The vertical line is the decision threshold: we reject H_0 for anything to its right.

The α region is the slice of the null bell to the right of the threshold — a true H_0 that we wrongly reject.
The β region is the slice of the alternative bell to the left of the threshold — a false H_0 that we wrongly keep.

The trade-off

The two errors pull against each other. Slide the threshold right to shrink \alpha (fewer false positives) and the \beta slice of the alternative bell grows — more misses, less power. Slide it left and the trade reverses. With fixed sample size you cannot drive both errors to zero at once; choosing \alpha is choosing where on this see-saw to sit. (Collecting more data is what separates the bells and eases the squeeze.)

Type I — reject a true H_0 (false positive), rate \alpha.
Type II — fail to reject a false H_0 (false negative), rate \beta.
Power = 1 - \beta — the chance of detecting a real effect.
Lowering \alpha raises \beta — a trade-off you can only escape with more data.

The tempting fix is "just make \alpha tiny so we never raise a false alarm." But look at the two bells: sliding the threshold right to shrink the \alpha slice automatically grows the \beta slice. Tightening against false positives inevitably creates more false negatives, and vice versa. With a fixed sample size there is no setting of the threshold that makes both errors small at the same time — it is a genuine see-saw, not a knob you can cheat.

The only escape is more data. A bigger sample pulls the two bells apart (their standard errors shrink), so a single threshold can sit in a comfortable gap with both \alpha and \beta small — that is what raising power means. This is why underpowered studies are so dangerous: with the bells still overlapping, they miss real effects and their occasional "significant" findings are more likely to be flukes. Small studies fail you in both directions at once.

The analogy is not loose — it is exact. A trial tests H_0 = "the defendant is innocent." Convicting an innocent person is a Type I error; freeing a guilty one is a Type II error. The principle "innocent until proven guilty," demanding proof "beyond reasonable doubt," is society deliberately setting a very small \alpha: we would rather almost never jail the innocent.

And that choice has a price, straight off the see-saw — a higher \beta: some guilty people go free. As the jurist William Blackstone put it, "better that ten guilty persons escape than that one innocent suffer." That is a value judgement about which error is worse, encoded as a threshold — the very same decision every experimenter makes when they pick \alpha.

Much of the cost is buying power. To reliably detect a real drug effect — to keep \beta low — a trial needs the two bells well separated, and the only way to get there is a large enough sample: thousands of patients, across many hospitals, over years. Run the trial too small and you are underpowered: even a genuinely effective drug may post a non-significant result purely by bad luck, and the entire multi-million study is wasted.