The Case for Throughput Processors

Around 2005, something broke that had held for thirty years: clock frequencies stopped rising. A 2004 desktop CPU ran at 3.8 GHz; a 2024 one runs at… roughly 5 GHz. Meanwhile the transistor budget kept right on doubling — from tens of millions to tens of billions. So chip architects faced a genuinely strange problem, the opposite of scarcity: an enormous and growing pile of transistors that could no longer buy single-thread speed. What do you spend them on?

The GPU is the most radical answer anyone gave: spend them on more, simpler cores — hundreds of them — and stop pretending any one thread matters. This lesson is about why that answer was not a stylistic choice but a corner the whole industry was backed into, and why, for the right class of work, the wimpy-cores bet wins on every metric that matters: operations per joule and operations per square millimetre.

Two walls, one corner

Two independent forces converged. First, the end of Dennard scaling: shrinking transistors stopped getting proportionally cheaper to switch, so cranking the frequency now cranked the power — and around 100 W per chip, air cooling says no. Frequency froze, and with it the free lunch of faster serial execution.

Second, the ILP wall: you can try to speed a single thread without frequency by executing more instructions per cycle, but the hardware that finds independent instructions — wider issue, bigger rename tables, deeper out-of-order windows, more bypass paths — grows roughly quadratically with issue width, while real programs offer stubbornly limited parallelism between nearby instructions. Each extra fraction of IPC costs more silicon and more energy than the last, for less return. By the mid-2000s the big out-of-order core was deep into diminishing returns.

Put the two together: transistors kept arriving (Moore continued for years after Dennard died), but neither frequency nor cleverness could convert them into serial speed. The only direction left was sideways: use the budget for parallel speed — many cores, each one small, simple, in-order, and cheap.

The workloads that were waiting

A sea of simple cores is useless unless work arrives in a matching shape: millions of independent, identical tasks. It turns out an enormous class of computing looks exactly like that:

Pixels — shade two million fragments per frame with the same lighting function; no pixel waits for another;
Matrix elements — every entry of a matrix product is the same dot-product recipe on different rows and columns;
Particles, rays, samples — physics simulation, ray tracing, Monte Carlo finance: the same small program over vast populations.

These are throughput workloads: nobody cares when any individual task finishes, only when all of them do. Latency per task is irrelevant; tasks per second is everything. For this shape of work, a processor optimised to make one thread fast is spending almost its entire area and energy on a property nobody asked for.

The iso-area tradeoff

Here is the tradeoff in one picture. Take a fixed die area and slice it into n equal cores. A useful rule of thumb (Pollack's rule) says a core's single-thread performance grows only as the square root of the area you lavish on it — that is the quadratic ILP tax read backwards. So each of the n small cores runs at \sqrt{1/n} the speed of one big core… but there are n of them, for a total throughput of n \cdot \sqrt{1/n} = \sqrt{n}:

Read both curves at n = 64: each core is 8\times slower, but the die as a whole does 8\times more work per second. If your workload has 64+ independent tasks, the right-hand end of this chart is free money. If it has one task, the right-hand end is a catastrophe. The entire CPU/GPU split falls out of which end of this chart your work lives on.

The energy arithmetic: tax versus work

The area argument has an even sharper energy twin. Ask what a big out-of-order core actually spends per instruction, and split it into work (the arithmetic you asked for) and tax (everything the core does to find, order and speculate around that arithmetic). Illustrative numbers for one 32-bit operation:

Energy sink	Big OoO core	Simple in-order lane
Fetch + decode (per instruction)	~20 pJ	shared across 32 lanes: ~1 pJ
Branch prediction, rename, scheduling	~25 pJ	— (in-order, no speculation)
Register file, bypass network	~15 pJ	~3 pJ (small, local)
The ALU operation itself	~5 pJ	~5 pJ
Total per operation	~65 pJ	~9 pJ

The punchline sits in the bottom-left cell: on a big core, the operation you wanted is about a tenth of the energy bill — the rest is overhead for making one stream go fast. A simple lane that fetches one instruction and applies it across 32 pieces of data amortises almost all of the tax away, landing at roughly 10–20× less energy per operation. Under a fixed power budget — which, post-Dennard, every chip lives under — energy per operation is throughput. The wimpy cores don't just fit more ops per mm²; they fit more ops per joule, and the joules are the binding constraint.

Amdahl's honesty clause

Before crowning the throughput machine, Amdahl's law demands its say. If a fraction p of a workload is parallelisable and the rest is stubbornly serial, then no matter how many cores you throw at it:

\text{speedup} = \frac{1}{(1-p) + \dfrac{p}{n}} \;\;\xrightarrow{\;n \to \infty\;}\;\; \frac{1}{1-p}

A workload that is 95% parallel caps at 20\times — forever — because the serial 5% ends up dominating. This is why the throughput processor joins the system rather than replacing it: real workloads are mixtures, and the serial fraction still needs a core that makes one thread fast. The modern platform is heterogeneous by necessity: a latency-optimised CPU for the serial spine, a throughput engine for the parallel bulk, and (as you saw with networks on chip) an interconnect stitching them into one system.

post-Dennard, frequency is capped by power; post-ILP-wall, serial cleverness costs quadratic silicon for diminishing IPC;
the growing transistor budget therefore buys parallel speed: many simple cores instead of one clever one;
for workloads of many independent identical tasks, simple lanes deliver ~10–20× more operations per joule (the OoO tax vanishes) and more operations per mm² (Pollack: throughput \propto \sqrt{n} at fixed area);
Amdahl's law keeps the CPU in the system: the serial fraction of a mixed workload still rules its total runtime.

In May 2004 Intel cancelled Tejas, the successor to the Pentium 4 — a core designed for a rumoured 7 GHz that, in prototype, ran so hot the roadmap collapsed under it. The company pivoted its entire line to multicore within two years; insiders called it "the right-hand turn." The Pentium 4's NetBurst pipeline was the last great monument to frequency-at-any-cost — over 30 pipeline stages, a design that only made sense if Dennard scaling kept paying. It didn't. The GPU makers, watching from the graphics side, realised their sea-of-simple-lanes design wasn't a niche adaptation for pixels — it was the general answer to the wall everyone had just hit. Three years later NVIDIA shipped CUDA, and "graphics card" quietly stopped being an accurate name.

The seductive error: "the GPU has 10,000 cores, so my program will run 10,000× faster." Port a sequential algorithm to a throughput processor and it runs on one wimpy lane — a lane deliberately built slower than a CPU core (no out-of-order window, no branch predictor worth the name, a modest clock). Your code gets slower, often dramatically. The throughput win exists only past thousands of independent work items — enough to fill the lanes and hide the stalls. One task: CPU. A dozen tasks: still CPU. A million tasks: now the GPU is unbeatable. Counting your independent work items is the first act of any porting decision, before a single line moves.