Benchmarking and the Right Mean
You know that the only honest measure of performance is
execution time
on a real workload. But which workload? No customer runs "a program" — they run their programs,
and you cannot ship every buyer a copy of your lab. So the industry agrees on a shared, public bundle of
representative programs — a benchmark suite — and reports one machine's time on each. The
canonical example is SPEC (the Standard Performance Evaluation Corporation): SPEC CPU
collects a few dozen real integer and floating-point programs (compilers, chess engines, fluid solvers,
video encoders) so that "SPEC scores" are at least measuring the same thing on every machine.
And here is where a beautiful, subtle trap springs shut. Once you have a dozen numbers, someone will want
one number — a single score to rank the machines. The moment you average them, the choice of
which average stops being a formality and starts changing the answer. Pick the wrong mean
and your summary will confidently lie.
The three means, and what each is for
Statistics gives you three "Pythagorean" means, and they are not interchangeable. For
n numbers x_1,\dots,x_n:
\text{AM} = \frac{1}{n}\sum_i x_i, \qquad \text{GM} = \left(\prod_i x_i\right)^{1/n}, \qquad \text{HM} = \frac{n}{\sum_i 1/x_i}.
They always satisfy \text{HM} \le \text{GM} \le \text{AM} (equal only when all
the numbers are identical). The rule of thumb that architects live by:
- Arithmetic mean — for quantities you would add up: raw
times. If you actually run all the programs once, total time is
\sum_i t_i and its average is the AM. Correct for absolute seconds.
- Harmonic mean — for rates (things per second: MFLOPS, IPC) when the
work is fixed. Averaging rates with the AM over-weights the fast cases; the HM is the honest
average rate.
- Geometric mean — for normalized ratios (speedups, "times relative to
a reference machine"). This is what SPEC reports.
Why the arithmetic mean of ratios lies
Suppose two benchmarks, and we normalize each machine's time to a reference machine (so a ratio
of 2 means "twice as slow as the reference"). Take these times:
| Program | Machine A | Machine B |
|---|
| P1 (time) | 10 s | 20 s |
| P2 (time) | 20 s | 10 s |
The two machines are obviously tied — each wins one program by the same factor, and the total
time is 30\,\text{s} for both. Now normalize the ratios to A (A becomes the
reference). A's ratios are 1, 1; B's ratios are
20/10 = 2 and 10/20 = 0.5.
- Arithmetic mean of B's ratios: (2 + 0.5)/2 = 1.25 — "B is
25\% slower!" But normalize to B instead and the same AM now declares
A is 25\% slower. A flat contradiction — the AM of ratios
depends on which machine you picked as reference.
- Geometric mean of B's ratios: \sqrt{2 \times 0.5} = 1. It
calls the tie correctly, and it gives the same ordering no matter which machine is the
reference. That reference-independence is precisely why SPEC uses it.
- average raw times (seconds) with the arithmetic mean;
- average rates (work / second, fixed work) with the harmonic mean;
- average normalized ratios / speedups with the geometric mean —
it alone is independent of the reference machine: \text{GM}(a_i/b_i) =
\text{GM}(a_i)/\text{GM}(b_i).
See the two means diverge
Take a machine that is 2\times faster on one benchmark and
8\times faster on another. What is its "average" speedup? The arithmetic mean
says 5\times; the geometric mean says 4\times. They
genuinely differ, and the gap grows as the ratios spread apart — the AM is dragged upward by the single
big number, which is exactly how a vendor "wins" by acing one friendly benchmark. Run it:
// Arithmetic vs geometric mean of a set of speedup RATIOS.
function arithMean(xs: number[]): number {
return xs.reduce((a, b) => a + b, 0) / xs.length;
}
function geoMean(xs: number[]): number {
const product = xs.reduce((a, b) => a * b, 1);
return Math.pow(product, 1 / xs.length);
}
const ratios = [2, 8]; // 2x faster on one benchmark, 8x on another
console.log(`ratios: ${ratios.join(", ")}`);
console.log(`arithmetic mean = ${arithMean(ratios).toFixed(3)} (inflated by the 8x)`);
console.log(`geometric mean = ${geoMean(ratios).toFixed(3)} (SPEC's choice)`);
// The AM of ratios also depends on the reference; the GM does not.
const A = [10, 20], B = [20, 10]; // times, two programs
const ratioBoverA = A.map((a, i) => B[i] / a); // normalize to A
const ratioAoverB = A.map((a, i) => a / B[i]); // normalize to B
console.log(`\nAM(B/A) = ${arithMean(ratioBoverA).toFixed(3)}, AM(A/B) = ${arithMean(ratioAoverB).toFixed(3)} -> both say the OTHER is slower (contradiction)`);
console.log(`GM(B/A) = ${geoMean(ratioBoverA).toFixed(3)}, GM(A/B) = ${geoMean(ratioAoverB).toFixed(3)} -> a consistent tie`);
Benchmarks are targets, and Goodhart's law bites: "when a measure becomes a target, it ceases to be a good
measure." Vendors have detected the exact benchmark binary and swapped in a hand-tuned code path;
compilers have recognised SPEC's specific loops and applied transformations that help only those
programs. Graphics drivers have rendered lower quality the instant they spotted a benchmark's window
title. None of it is illegal, and all of it makes the score meaningless. This is why SPEC continually
retires and rotates its programs (CPU2000 → CPU2006 → CPU2017), forbids benchmark-specific flags
in its "base" scores, and requires reproducible, documented runs. A single number is always attackable;
the defence is a broad, fresh, transparent suite.
The deepest error is not choosing the wrong mean — it is trusting any single number. A machine
with a huge cache flies on programs that fit in it and crawls on those that don't; a wide vector unit
doubles floating-point throughput and does nothing for a branchy compiler. The geometric mean is the
least-bad summary of a suite, but it still hides the spread. Always look at the whole
distribution of per-benchmark results, and match the suite to your workload — a
database server should not be bought on a ray-tracing score. The right mean protects you from one lie; only
the full data protects you from the rest.
The professional habit
So: measure real programs, report them individually, and when you must collapse to one number, pick the
mean that matches the quantity — arithmetic for times, harmonic for rates, geometric for
ratios. Then remember that even the right single number is a summary of a summary. The next
lessons look at why machines got fast enough to need such careful measurement in the first place —
the exponential engines of
Moore's law and Dennard scaling.