A cereal company prints "500 g" on every box. You weigh 12 boxes and get an average of 496 g. Is the factory short-changing you — or is 496 just the ordinary wobble you'd expect from weighing only a dozen boxes? A juice trial reports the new formula lowered blood pressure by 3 mmHg on average. Real drug, or lucky noise? Almost every question of this shape — "is this sample mean far enough from a claimed value (or from another group's mean) to be more than random chance?" — is answered by the same everyday workhorse: the t-test.

It is, quietly, the single most-used tool in quantitative science. A huge fraction of the numbers you'll ever read in a paper, a medical label, or an A/B test dashboard came out of a t-test. This page shows you exactly what it computes, and — just as important — what its famous verdict "statistically significant" does and does not mean.

The significance level α: a threshold chosen in advance

How small must a p-value be before we reject H_0? Rather than argue case by case, we fix a threshold in advance: the significance level \alpha. The rule is then mechanical,

with \alpha = 0.05 the most common choice. Committing to \alpha before seeing the data stops us from moving the goalposts to fit the result. \alpha is also the rate at which we would wrongly reject a true H_0 — a deliberately chosen risk. Pick \alpha = 0.05 and you accept that, in a world where nothing is really going on, you'll still cry "effect!" about 1 time in 20.

The t-statistic

To test a mean we need a statistic. Standardising the sample mean gives the t-statistic

It is just a z-score for the sample mean: how many standard errors the observed mean \bar{x} sits from the value \mu_0 claimed by H_0. Read the fraction as signal over noise: the top is how far your data drifted from the claim, the bottom is how much drift pure chance routinely produces. A big |t| means the gap dwarfs the noise; a small one means the gap is well within what randomness serves up. The one twist is the s in the denominator: we rarely know the true \sigma, so we use the sample's own standard deviation s as a stand-in.

Worked example 1 — the cereal boxes

Back to the "500 g" boxes. We weigh n = 16 of them and find a sample mean \bar{x} = 496 g with sample standard deviation s = 8 g. The null claim is \mu_0 = 500. Step by step:

The sample mean sits two standard errors below the printed weight. With \nu = n - 1 = 15 degrees of freedom, a two-sided |t| = 2.0 gives a p-value of about 0.064 — just above 0.05. So at \alpha = 0.05 we would not reject H_0: the shortfall, while suggestive, is still within reach of ordinary sampling noise for 16 boxes. Weigh more boxes and the same 4 g gap could easily tip over the line.

Worked example 2 — a reaction-time claim

A trainer claims their drill leaves athletes with a mean reaction time of \mu_0 = 250 ms. You test n = 25 athletes and record \bar{x} = 240 ms with s = 20 ms.

Here |t| = 2.5 with \nu = 24 degrees of freedom gives a two-sided p-value near 0.02, comfortably below 0.05, so we reject H_0: this looks like a real difference from the claimed 250 ms. Notice the pattern — a larger |t| and a larger n both push you toward significance.

One sample vs two samples

Both examples above are one-sample t-tests: is this group's mean different from a single fixed claim \mu_0? Often, though, there is no printed claim — instead you have two groups and want to know if they differ from each other: treatment vs placebo, layout A vs layout B, this year's crop vs last year's. That is a two-sample t-test, and it keeps the very same shape,

where the numerator is now the difference between the two group means and the denominator is the standard error of that difference (built from both groups' spreads and sizes). Conceptually it is identical: an observed difference divided by the noise you'd expect in that difference by chance. One-sample asks "far from a claim?"; two-sample asks "far from each other?".

Why the tails get heavier

Using s in place of \sigma has a price. Because s is itself an estimate — and a jumpy one when n is small — the statistic t scatters more widely than a true z-score would. So t does not follow the standard normal; it follows the t-distribution, which is bell-shaped and symmetric but has heavier tails — more probability stranded far from the centre.

The t-distribution is indexed by its degrees of freedom \nu = n - 1. Small \nu means a small, noisy sample and the fattest tails; as \nu grows the estimate s settles down and the t-distribution approaches the standard normal. Slide the degrees of freedom and watch the heavy-tailed curve tighten onto the bell.

Putting it together

A t-test: compute t = (\bar{x}-\mu_0)/\mathrm{SE}, find its p-value from the t-distribution with n-1 degrees of freedom, and reject H_0 when p < \alpha. For large n the t- and z-tests nearly coincide; for small n the heavier t-tails demand a more extreme t before they will let you reject — a built-in penalty for small data. And remember what a test can be wrong about in two opposite ways: the verdict is never a guarantee.

See it explained