Analysis of Variance (ANOVA)

A school tries three ways to teach the same topic — call them methods A, B and C — then gives every class the same test. Method A averages 10 out of 20, B averages 13, C averages 16. The averages differ, sure — but each class also has its own internal wobble: strong and weak pupils, a good day, a bad day. So the real question is not "are the three averages identical?" (they never are) but "are they farther apart than the ordinary wobble inside each class would explain?"

With two groups you already know the tool — the t-test. But now there are three, and soon there might be five or ten (five fertilisers, six drug doses, eight website layouts). Running a separate t-test on every pair quietly breaks — for reasons we'll see. The one clean tool for "are these k group means all equal?" is Analysis of Variance, universally shortened to ANOVA. Its trick is beautiful and a little surprising: to compare a bunch of means, it compares variances.

The null hypothesis: all the means are equal

One-way ANOVA tests a single, blunt claim. With k groups it puts on trial

H_0:\ \mu_1 = \mu_2 = \cdots = \mu_k \qquad\text{(every group has the same true mean),}

against the alternative that at least one group mean is different from the rest. Notice how weak the alternative is: it does not say which group, or how many, or in which direction — only that the flat "everyone's equal" picture fails somewhere. Holding on to that modesty will save you from the single most common misreading of an ANOVA result.

The big idea: between-group vs within-group spread

Picture the pupils' scores as dots. Two very different kinds of variation live in that picture:

If the methods truly do nothing, the group means will still drift apart a little just by luck — but only about as much as the within-group noise would predict. If a method really works, the group means fly apart faster than that noise can explain. So the whole test boils down to a single ratio,

F = \frac{\text{variation between the group means}}{\text{variation within the groups}} = \frac{\text{signal}}{\text{noise}}.

A large F means the gaps between groups dwarf the wobble inside them — evidence the means genuinely differ. An F near 1 means the between-group gaps are no bigger than the everyday within-group churn — nothing to see here.

Partitioning the sum of squares

To turn that idea into arithmetic we split the total spread of all the data into the two pieces above. Take every data point x_{ij} (point j in group i), the grand mean \bar{x}, and each group mean \bar{x}_i. The variance-style total sum of squares splits exactly in two:

\underbrace{\sum_{i}\sum_{j}(x_{ij}-\bar{x})^2}_{\text{SST (total)}} \;=\; \underbrace{\sum_{i} n_i(\bar{x}_i-\bar{x})^2}_{\text{SSB (between)}} \;+\; \underbrace{\sum_{i}\sum_{j}(x_{ij}-\bar{x}_i)^2}_{\text{SSW (within)}}.

In words: SST = SSB + SSW. Total wobble equals between-group wobble plus within-group wobble, with nothing left over — a tidy Pythagorean-style identity. SSB measures how far each group mean strays from the grand mean (weighted by group size n_i); SSW is just the pooled spread inside the groups.

From sums of squares to the F-statistic

A raw sum of squares grows simply because you added more numbers, so before comparing SSB with SSW we divide each by its degrees of freedom to get an average — a mean square. With k groups and N data points in total:

\text{MSB} = \frac{\text{SSB}}{k-1}, \qquad \text{MSW} = \frac{\text{SSW}}{N-k}.

There are k-1 degrees of freedom between groups (k means, minus one for the grand mean they share) and N-k within (N points, minus one mean estimated per group). The test statistic is their ratio:

F = \frac{\text{MSB}}{\text{MSW}}.

Under H_0 both mean squares estimate the same underlying noise variance, so F hovers around 1. When H_0 is false, MSB picks up the extra between-group signal and swells, pushing F well above 1. We compare the observed F against the F-distribution with (k-1,\ N-k) degrees of freedom; a value out in its upper tail (small p-value) means reject H_0.

Worked example 1 — the three teaching methods

Three pupils sit each method's test (small numbers so we can do it by hand). Scores out of 20:

\begin{array}{c|ccc|c} \text{Method} & \multicolumn{3}{c|}{\text{scores}} & \bar{x}_i \\ \hline A & 8 & 10 & 12 & 10 \\ B & 11 & 13 & 15 & 13 \\ C & 14 & 16 & 18 & 16 \end{array}

Here k = 3 groups, n_i = 3 each, so N = 9. The grand mean is \bar{x} = (10+13+16)/3 = 13.

Within groups (SSW). Inside every method the scores sit at -2, 0, +2 around their own mean, giving (-2)^2+0^2+2^2 = 8 per group:

\text{SSW} = 8 + 8 + 8 = 24, \qquad \text{MSW} = \frac{\text{SSW}}{N-k} = \frac{24}{9-3} = \frac{24}{6} = 4.

Between groups (SSB). The group means sit at -3, 0, +3 around the grand mean 13, each weighted by n_i = 3:

\text{SSB} = 3\big[(-3)^2 + 0^2 + (3)^2\big] = 3(18) = 54, \qquad \text{MSB} = \frac{\text{SSB}}{k-1} = \frac{54}{2} = 27.

The F-statistic.

F = \frac{\text{MSB}}{\text{MSW}} = \frac{27}{4} = 6.75, \qquad (k-1,\ N-k) = (2,\ 6).

The critical value of the F-distribution with (2,6) degrees of freedom at \alpha = 0.05 is about 5.14. Our F = 6.75 clears it, so we reject H_0: the methods are not all equal — the between-group gaps are too big to blame on ordinary pupil-to-pupil noise. (Quick sanity check on the partition: \text{SST} = \text{SSB} + \text{SSW} = 54 + 24 = 78, which is exactly the total squared spread of all nine scores about 13.)

Worked example 2 — why not just run t-tests on every pair?

Faced with three groups, the tempting shortcut is: t-test A vs B, A vs C, and B vs C — three tests, done. The problem is that error piles up. Each t-test at \alpha = 0.05 carries a 5% chance of a false alarm (crying "difference!" when there is none). Run several and the chance that at least one raises a false alarm grows. If the tests were independent, the probability that all three come back correctly "not significant" when nothing is really going on is

(1 - 0.05)^3 = 0.95^3 \approx 0.857,

so the chance of at least one false positive is

1 - 0.857 = 0.143 \approx 14\%,

nearly three times the 5% you thought you signed up for. With k = 5 groups there are \binom{5}{2} = 10 pairwise tests and the family-wise error rate balloons to 1 - 0.95^{10} \approx 0.40 — a 40% chance of a spurious "finding". ANOVA sidesteps the whole mess: it asks the single question "are they all equal?" in one test at one honest \alpha, keeping the error rate exactly where you set it.

See it: watch F respond to the spread

Three groups of dots (one colour each), with a thick dash marking each group mean and a dashed line for the grand mean. The scatter inside each column — the within-group noise, and so SSW — never changes. Drag the slider to pull the group means apart: only the between-group spread (SSB) grows, and the live F = \text{MSB}/\text{MSW} climbs with it.

Slide it to the left and the three clouds overlap into one blur: the means huddle at the grand mean, F collapses toward 0, and no reasonable test would call the groups different. Slide it right and the clouds separate cleanly: F shoots past the \approx 3.89 cutoff for (2,12) at \alpha = 0.05 and the difference becomes "significant". Same noise throughout — only the signal moved.

Putting it together

ANOVA compares the spread between group means to the spread within groups. If the first dwarfs the second, the means are probably not all equal.

This is the classic ANOVA trap. A significant result tells you only that the flat "all equal" picture failed somewhere — not which group is the odd one out, nor how many differ, nor in which direction. To pinpoint the culprits you follow up with a post-hoc test (Tukey's HSD, Bonferroni-corrected pairwise tests, and friends) that compares pairs while controlling the error rate you were trying to protect in the first place.

Three more things to keep straight. F is a ratio: it's between-over-within, so a big SSB alone proves nothing — it's only big relative to the noise that counts, and a modest gap over tiny noise beats a huge gap over huge noise. ANOVA also leans on assumptions: roughly equal group variances (homogeneity) and roughly normal data within groups; wildly unequal spreads or heavy skew can mislead it, and then a different test (Welch's ANOVA, Kruskal–Wallis) is safer. And F is never negative — it's a ratio of squared quantities, so there are no "left-tail" surprises.

The whole machinery is the invention of Ronald A. Fisher, who in the 1920s was working at the Rothamsted agricultural station in England, drowning in field trials: which of a dozen fertilisers, barley strains, or manures gave the best yield across plots that varied wildly on their own? Comparing every pair by hand was hopeless and, as we've seen, statistically rotten. Fisher's leap was to partition the variance of the whole experiment into named, additive chunks — treatment vs error — and read the story from their ratio.

That ratio was later christened F in his honour (by George Snedecor), and the same idea grew into the entire field of experimental design: randomisation, blocking, factorial experiments. Modern science's habit of running controlled, multi-group experiments and letting an F-statistic arbitrate traces straight back to a statistician trying to grow better crops.