Latency vs Throughput Design

Give two architects the same transistor budget and the same physics, and ask each to build a processor. One returns a CPU core; the other returns a GPU's streaming multiprocessor. Neither made a mistake — they answered different questions. The CPU architect was asked "how fast can you finish one thread?" and the GPU architect "how much total work can you finish per second, given millions of threads?" — the question the last lesson showed the industry being forced to ask.

This lesson is about how differently those two questions spend the same silicon, and about the single deepest consequence: the two designs handle the enemy — memory latency — with opposite strategies. One hides it by cleverness. The other hides it by multitude.

Cleverness: the CPU's spend

A modern CPU core is mostly not arithmetic. Walk its die and you find: a large cache hierarchy (so most loads never pay DRAM's price), a branch predictor (so the pipeline never waits to learn where the program goes), an out-of-order window with rename registers (so independent instructions slip past a stalled one), and speculation machinery everywhere. Every block exists to serve one thread's latency: to make the gap between "instruction issued" and "result ready" invisible to a single stream of execution. The actual ALUs are a sliver in the corner.

Multitude: the GPU's spend

The GPU architect deletes nearly all of that and spends the freed area on two things: lanes (row upon row of ALUs) and thread contexts (an enormous register file holding the live state of thousands of resident threads at once). The latency-hiding strategy becomes brutally simple: when a thread waits on memory, run a different one. The stalled warp's registers stay put in the register file; the scheduler just issues from another warp that is ready. Nothing was predicted, nothing was reordered, and — crucially — the latency did not shrink by a single nanosecond. It got covered, buried under other threads' useful arithmetic.

The same area, drawn to scale

Here are the two philosophies as floor plans. Step through and watch where the area goes — the blocks that dominate one die barely exist on the other:

Worked example: how many warps to bury a load?

Make the multitude strategy quantitative. Suppose a DRAM access costs 400 cycles, and a running warp issues a new instruction every 4 cycles. When one warp fires off a load and stalls, how many other warps must be resident so the ALUs never notice?

\text{warps needed} \;=\; \frac{\text{memory latency}}{\text{issue interval}} \;=\; \frac{400}{4} \;=\; 100

One hundred resident warps — at 32 threads each, that is 3200 live threads per SM, times maybe a hundred SMs across the chip. This is the origin of the strangest fact about GPUs: they don't merely tolerate being flooded with threads, they require it. Oversubscription — far more threads than lanes — is a design assumption, not an accident. The fraction of those warp slots you actually manage to fill has a name, occupancy, and measuring it is a lesson of its own at the end of this module. With too few warps, every cycle of that 400-cycle latency shows through as idle silicon.

The cache philosophy flips

Deleting the big caches sounds reckless until you see what the GPU's small ones are for. On a CPU, a cache exists to save one thread from waiting: hit rate is everything, and a miss is a small catastrophe. On a GPU, misses are expected — thousands of threads streaming over huge arrays cannot possibly fit in any cache — and a miss is no catastrophe at all, because warp-switching covers it. The GPU cache's real job is to be a bandwidth filter: catch the re-used fraction of traffic close to the lanes so the precious DRAM interface only carries what it must. A CPU cache protects latency; a GPU cache protects bandwidth. Same SRAM, opposite job description.

The ancestor: vector machines

None of this appeared from nowhere. The lane half of the design is the vector processor reborn: one instruction commanding an array of ALUs over an array of data, amortising fetch and decode across the width. What the GPU adds is the second ingredient — massive multithreading — so that the vector lanes never sit idle waiting for memory. A fair one-line genealogy: a GPU is a vector machine crossed with a thread-switching machine, Cray's lanes wearing a scheduler that always has somewhere else to be.

a latency design spends area making one thread fast: caches, prediction, out-of-order speculation — hiding stalls by cleverness;
a throughput design spends area on lanes and thread contexts — hiding stalls by multitude: when a thread waits, run another;
covering a latency of L cycles with warps issuing every i cycles needs about L/i resident warps — so oversubscription is a requirement, not a luxury;
cache roles flip: CPU caches protect latency (hit rate is everything); GPU caches are bandwidth filters (misses are expected and covered).

Latency-hiding by thread-switching is older than most people guess. Seymour Cray's CDC 6600 (1964!) had ten peripheral processors that were secretly one processor rotating through ten thread contexts like a barrel — a "barrel processor". The idea reached its purest form in the Tera MTA (1990s): 128 hardware threads per core, a new thread issued every single cycle, and — the audacious part — no data cache at all. Why bother, when you can always switch to a thread that isn't waiting? The MTA was commercially doomed (too exotic, too few threads in 1990s software), but it was simply early. Twenty years later the same bet, welded onto vector lanes and fed by graphics workloads with millions of threads, conquered computing. In architecture, being wrong is often just being early with the workload missing.

A persistent myth says CPUs have low memory latency and GPUs high. In truth a DRAM access costs roughly the same few hundred nanoseconds for both — it's the same physics and much the same DRAM. The difference is entirely in the strategy: the CPU tries to avoid the latency (caches, prediction) and the GPU to tolerate it (warp-switching). The myth turns dangerous when you under-fill a GPU: launch a kernel with too few threads and there is no multitude to hide behind — every nanosecond of DRAM latency lands directly on your runtime, on a machine with no cleverness to fall back on. A starved GPU exposes latency more brutally than any CPU would.