Measuring Performance

"Which computer is faster?" sounds like a simple question. It is a trap. Faster at what, and faster for whom? A delivery van and a sports car are both "fast", but a courier moving ten thousand parcels a day does not want the sports car. Before you optimise anything — before you touch a pipeline, a cache, or a clock — you must decide which number you are trying to make bigger. Architecture is an engineering discipline, and engineering begins with a defined metric.

There are two headline metrics, and they are not the same thing:

Latency (a.k.a. response time, execution time) — how long one task takes, start to finish. Measured in seconds. Lower is better.
Throughput (a.k.a. bandwidth) — how many tasks finish per unit time. Measured in tasks/second (or bytes/second, instructions/second…). Higher is better.

The sports car wins on latency (one trip, done quickly); the van wins on throughput (parcels delivered per hour). A data-centre operator sells throughput; a gamer chasing frame-times buys latency. Same hardware, opposite priorities.

Performance is one over time

To make "faster" precise, architects define performance as the reciprocal of execution time for the task you care about:

\text{Performance} \;=\; \frac{1}{\text{execution time}}.

Then "machine X is faster than machine Y" means exactly \text{Performance}_X > \text{Performance}_Y, i.e. \text{time}_X < \text{time}_Y. The amount is the speedup:

\text{Speedup} \;=\; \frac{\text{Performance}_X}{\text{Performance}_Y} \;=\; \frac{\text{time}_Y}{\text{time}_X}.

Notice the flip: performance is a ratio of times the other way round. If X runs a job in 2\,\text{s} and Y takes 3\,\text{s}, the speedup of X over Y is 3/2 = 1.5\times — X is 50\% faster, not 33\%. Getting this ratio upside-down is the single most common error in performance reports.

\text{Performance} = 1/\text{execution time} — bigger is faster;
speedup of X over Y is \dfrac{\text{time}_Y}{\text{time}_X} (times divide the opposite way to performances);
latency is time-per-task (seconds, lower better); throughput = bandwidth is tasks-per-second (higher better) — improving one need not improve the other.

The metric zoo

The words pile up fast, and vendors exploit the confusion. Here is the vocabulary, pinned down:

Metric	Units	Means	You care when…
Latency / response time	seconds	time for one task, end to end	a single request must feel instant
Throughput / bandwidth	tasks/s, GB/s	work completed per unit time	you serve millions of requests
Wall-clock time (elapsed)	seconds	real time you actually waited	you measure the whole experience
CPU time	seconds	time the CPU spent on your program	you compare processors fairly

The last two are subtly different and matter enormously. Wall-clock time is what a stopwatch reads: it includes waiting on the disk, the network, and every other program the OS was running. CPU time is only the seconds the processor actually spent executing your code (often split into user and system CPU time). A job that "took 10 seconds" on the stopwatch but used only 2\,\text{s} of CPU was mostly waiting — and buying a faster CPU will barely help it. Always know which clock you are reading.

Latency and throughput are different knobs

The cleanest way to feel the difference is pipelining — a laundromat, or a CPU. Suppose one task passes through k equal stages and each task takes total latency L = 1 unit end to end. Run tasks one at a time (unpipelined) and N tasks cost N \cdot L. Overlap them in a k-stage pipeline and, once it is full, a new task pops out every L/k — so

T_{\text{pipe}}(N) \;=\; L \;+\; (N-1)\,\frac{L}{k}.

Drag the depth slider below. The first task still takes the full latency L — deep pipelining does not make a single task finish sooner. But the slope flattens: throughput (tasks per unit time, the inverse slope) climbs toward k\times. That is the headline lesson of this whole course — most tricks buy throughput, and latency is the stubborn one.

Worked example — read the speedup carefully

Machine A renders a frame in 25\,\text{ms}; a redesigned machine B renders it in 20\,\text{ms}. Speedup of B over A:

\text{Speedup} = \frac{\text{time}_A}{\text{time}_B} = \frac{25}{20} = 1.25\times \;\;\Rightarrow\;\; B \text{ is } 25\% \text{ faster}.

For throughput, invert: A does 1000/25 = 40 frames/s, B does 1000/20 = 50 frames/s — the same 1.25\times, because for a single stream of identical tasks throughput really is 1/\text{latency}. The two metrics only diverge once you overlap or parallelise tasks (as in the pipeline above). Run the code to see both readings computed from raw times.

// Performance = 1 / execution time; speedup of X over Y = time_Y / time_X. function throughput(msPerTask: number): number { return 1000 / msPerTask; // tasks per second } function speedup(timeSlow: number, timeFast: number): number { return timeSlow / timeFast; } const tA = 25, tB = 20; // milliseconds per frame console.log(`A: ${throughput(tA).toFixed(1)} fps B: ${throughput(tB).toFixed(1)} fps`); console.log(`speedup B over A = ${speedup(tA, tB).toFixed(3)}x`); console.log(`B is ${((speedup(tA, tB) - 1) * 100).toFixed(1)}% faster`); // Pipelining: latency of the FIRST task is unchanged; throughput rises toward k times. const L = 12, k = 4; // 12 ns latency, 4 equal stages const perTask = L / k; // steady-state issue interval once the pipe is full console.log(`first task still takes ${L} ns; steady-state throughput = ${(1000 / perTask).toFixed(0)} M-tasks/s`);

Famously: "never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." It is a genuine engineering point, not just a joke. A car carrying a few hundred hard drives across a city moves petabytes — divide by the drive time and the throughput (bytes per second) can beat any internet link. But the latency is dreadful: the first byte does not arrive for hours. This is exactly why cloud providers really do ship crates of disks (Amazon's "Snowball") to move giant datasets: when you need throughput and can tolerate latency, physical shipping wins. Latency and throughput are genuinely independent axes.

Vendors love single-number badges — megahertz, MIPS (millions of instructions per second), FLOPS. They are seductive and misleading. MIPS counts instructions, but instructions differ between ISAs and programs, so a chip can post huge MIPS on trivial instructions while doing little real work — some old-timers expanded MIPS as "Meaningless Indicator of Processor Speed". Clock rate (GHz) is only one of the three factors in the iron law. The only honest performance metric is execution time on the workload you actually run — everything else is a proxy that can be gamed.

The discipline: define the metric first

Every optimisation in this course improves some metric on some workload. Widening the pipeline lifts throughput but may raise the latency of a branch. A bigger cache cuts average latency but costs area and power. There is no scalar "speed"; there is only "faster at this task, measured this way". So the professional habit is: state the workload, state the metric, then measure execution time — and only then reach for the benchmark suites and the right average that let you compare machines without fooling yourself.