The p-Value
The p-value is the single most used — and most misused — number in all
of science. It appears in nearly every research paper, decides which drugs reach patients, and
launches a thousand headlines a week. And yet it answers exactly one narrow
question, no more:
- If the null hypothesis were true, how likely would it be to see data at least this
extreme, purely by chance?
A small p-value means "this data would be surprising if nothing were really going
on" — and that surprise is exactly what counts as evidence against the null. A
large p-value means the data are the sort of thing chance coughs up all the time, so there is
nothing to get excited about.
A hypothesis test
leaves us with a vague feeling that the data are "surprising" or "not surprising" under
H_0. The p-value turns that feeling into a number.
It is the probability — computed as if H_0 were true — of
getting data at least as extreme as what we actually saw:
p = \mathbb{P}\bigl(\text{data this extreme, or more} \mid H_0\bigr).
Everything hangs on the conditioning bar: we live entirely inside the world where
H_0 holds, and ask how often that world would throw up a
result as unusual as ours.
Small p, big surprise
Read the size of p directly as surprise:
-
a small p (say 0.01)
means data this extreme would be rare under H_0 — so the data are
evidence against H_0;
-
a large p (say 0.40)
means data this extreme are quite ordinary under H_0 — no reason
to doubt it.
Geometrically, p is an area in the tail of the null
distribution: the total probability of outcomes at least as far from the centre as the one we
observed.
The p-value is the shaded tail
Here is the null sampling distribution with an observed statistic marked at
z = 1.6. The shaded region beyond it is the p-value: the chance,
under H_0, of landing at least that far out. The smaller that sliver
of area, the more the data strain against H_0.
(This picture shades only the upper tail, a one-sided p-value. For a two-sided
H_1: \mu \ne \mu_0 we would shade both tails, since "more extreme"
runs in either direction.)
Reading a p-value out loud
The whole point of a p-value is that you can translate it into a plain English sentence. The
template is always the same: "if the null were true, we'd see a result this strong only
{p} of the time by luck." Try it:
-
p = 0.03 for a drug trial: "if the drug did nothing,
we'd see a result this strong only 3% of the time by chance." Fairly
surprising — decent evidence the drug does something.
-
p = 0.001: "a result this strong would happen only
1 in 1000 times if nothing were going on." Very strong evidence against
H_0 — this is hard to wave away as luck.
-
p = 0.20: "a result this strong turns up
1 in 5 times by pure chance." That happens constantly — the data are entirely
consistent with H_0, and we have found nothing.
Notice how p = 0.001 and p = 0.20 tell
opposite stories, even though both are "just a p-value". The number is a dial of surprise,
not an on/off switch.
Beware the magic line at 0.05
By long convention, p < 0.05 gets stamped
"statistically significant" and we reject H_0. But
0.05 is a completely arbitrary threshold — a round
number a statistician liked in the 1920s, not a law of nature. (Where the line
should sit, and how the t-test computes p in the first place,
is the story of
significance and the t-test.)
Because the cutoff is arbitrary, the difference between the two sides of it is meaningless. Consider
p = 0.049 and p = 0.051:
- p = 0.049 → "significant!", publish, celebrate;
- p = 0.051 → "not significant", shelve the study.
Yet these two numbers describe essentially identical evidence — the data are a
hair's breadth apart. Treating one as a triumph and the other as a failure is one of the silliest,
and most common, habits in applied statistics. A p-value is a smooth measure of surprise; the
0.05 line chops that smooth dial into a fake yes/no.
The misreading that will not die
The p-value is not the probability that H_0 is true.
It is computed assuming H_0 — so it can say nothing about
H_0's own probability. The conditioning runs
\mathbb{P}(\text{data}\mid H_0), never
\mathbb{P}(H_0\mid\text{data}); swapping the two is exactly the error
Bayes' theorem warns about. A p of 0.03 does not mean "a 3% chance
H_0 is true".
- p = \mathbb{P}(\text{data at least this extreme}\mid H_0) — a tail area of the null distribution.
- Small p ⇒ the data would be surprising under H_0 ⇒ evidence against H_0.
- p is not \mathbb{P}(H_0\text{ is true}) — it conditions on H_0, it does not measure it.
The p-value is astonishingly easy to misquote. Two errors dominate:
-
It is not the probability that the null is true, and not "the probability your
result was due to chance". It is \mathbb{P}(\text{data}\mid H_0) — the
probability of the data given the null — not
\mathbb{P}(H_0\mid\text{data}), the probability of the null given the
data. Reversing a conditional probability like this is arguably the single most common
statistical error in existence. (It is the same slip as confusing "most terrorists are men" with
"most men are terrorists".)
-
A small p says nothing about whether the effect is big or important. With a huge
sample, a laughably tiny effect — a drug that lowers blood pressure by
0.1 of a unit — can produce p = 0.0001. The
p-value tells you the effect is detectable, never that it matters. For that you
need the size of the effect, which the p-value flatly refuses to report.
Suppose H_0 is true and you run one test: there is a
5\% chance of a false "significant" result. Now run twenty
tests on the same data — different subgroups, different outcomes, different ways of slicing it — and
the chance that at least one crosses p < 0.05 by pure luck
climbs to about 64\%. Report only that one, and you have "found" an
effect that does not exist.
That practice — often unconscious — is p-hacking: trying analyses until one
crosses the magic line, then telling the story as if you had asked that question all along. It is
why a study "showing" that some everyday food causes (or prevents) cancer surfaces almost every
week, grabs a headline, and then quietly fails to replicate.
This got serious enough that in 2016 the American Statistical Association issued a
rare formal statement warning against exactly this misuse of p-values, and some journals have gone
as far as banning them outright. It is a genuine, ongoing crisis in how science is
done — and it starts with treating a single number as a verdict instead of a clue.
See it explained