GPU Caches and the Memory Hierarchy

Yes, GPUs have caches — and if you carry your CPU cache intuition across unedited, they will lie to you. On a CPU, the cache's job is to make one thread's memory look fast: keep its working set close, and a load that would cost 300 cycles costs 4. A GPU cache serves a hundred thousand threads at once, and split a hundred thousand ways there is no "close" left to offer anyone. So GPU caches do a different job — not hiding latency (warps and the scheduler do that) but filtering bandwidth: absorbing re-fetches so that precious DRAM bandwidth is spent only on bytes that truly have to come from DRAM. Same SRAM, same tag arrays, completely different purpose. This lesson rebuilds your mental model of the hierarchy, GPU-style.

The hierarchy, top to bottom

Four levels, each roughly an order of magnitude bigger and slower than the last:

Level	Size (representative)	Visible to	Latency (approx.)
Register file	256 KB per SM	one thread (its own registers)	~1 cycle
Shared memory / L1	128 KB per SM (one SRAM pool)	block (shared) / SM (L1)	~20–30 cycles
L2 cache	a few MB, in slices	the whole chip	~200 cycles
GDDR / HBM DRAM	tens of GB	everything	~400–600 cycles

The shapes are CPU-familiar; the per-thread arithmetic is not. Do it once and the whole philosophy of GPU caching falls out. A mainstream part has, say, 2 MB of L2 shared by roughly 100,000 resident threads:

\frac{2\,\text{MB}}{100{,}000 \text{ threads}} \approx 20 \text{ bytes per thread.}

Twenty bytes. Not twenty kilobytes — twenty bytes, five floats, per thread. The per-SM L1 fares little better: 128 KB across up to 2,048 resident threads is 64 B each. And now the punchline of the whole hierarchy: the register file, at 256 KB per SM, is bigger than the L1 and gives each thread 128 B of truly private, single-cycle storage. Per thread, the GPU pyramid is upside down: registers are the roomiest fast storage a thread will ever see, and each cache level below offers it less. Compare a CPU core: 32 KB of L1 split between 2 SMT threads is 16 KB each — about a thousand times the GPU figure.

One picture of the imbalance

Plotted per thread — the only fair way to compare machines with such different thread counts — the hierarchy looks like this:

This chart is why the rest of the lesson keeps repeating one sentence: on a GPU, latency is hidden by warps, and caches are for saving bandwidth.

What a bandwidth filter actually does

If a cache cannot hold anyone's working set, what is it for? Catching sharing and spillover — the reuse that exists between threads rather than within one:

Cross-thread reuse. Thousands of threads read the same lookup table, the same matrix tile edge, the same neighbour cells of a stencil. The first reader pays DRAM; the L2 serves everyone else. DRAM sees one fetch instead of thousands — bandwidth filtered.
Coalescing's safety net. When a warp's accesses are imperfect — misaligned or slightly scattered — caches with sectored lines (128-byte lines fetched as four independent 32-byte sectors) hold the over-fetched sectors, so a neighbouring warp that wants them a moment later finds them without a second trip to DRAM.
Traffic shaping, on request. Modern GPUs let code annotate accesses: a streaming hint marks data that will not be reused (evict it first — don't let it flush the useful stuff), and an L2 residency window can pin a hot region — that one lookup table every thread hammers — so it never leaves the L2 at all.

Notice what is absent: no per-thread latency story. A hit in the GPU's L2 still costs ~200 cycles — useless for making one thread feel fast, which is fine, because no one is asking it to. Ready warps cover the 200 cycles; the L2's contribution is that DRAM's bandwidth wasn't spent.

One honest paragraph about coherence, because CPU habits expect it: the per-SM L1s are not kept coherent with each other. If SM 3 writes a value that SM 40 then reads from its own stale L1, the hardware will not fix it for you. The discipline is simple and GPU-shaped: threads in different blocks communicate through global memory only at kernel boundaries (a new kernel launch invalidates the L1s and sees everything), or via atomics, which operate at the L2 — the one point every SM agrees on. Writes typically go through the L1 (write-through, no write-allocate) to the L2 precisely so that the little lies the L1s tell stay read-only and short-lived.

Thrash, executable

The per-thread arithmetic predicts something testable: a working set that a lone thread could cache comfortably becomes uncacheable when ten thousand siblings share the same SRAM. One tiny direct-mapped cache, two worlds:

// One cache, two worlds: a lone thread with reuse vs a crowd sharing the same SRAM. const LINES = 1024; // cache capacity, in lines function hitRate(stream: number[]): number { const tag = new Array(LINES).fill(-1); // direct-mapped: slot = line % LINES let hits = 0; for (const line of stream) { const slot = line % LINES; if (tag[slot] === line) hits++; else tag[slot] = line; } return hits / stream.length; } // World 1: ONE thread sweeps its 256-line working set ten times (a CPU-ish loop). const alone: number[] = []; for (let pass = 0; pass < 10; pass++) for (let i = 0; i < 256; i++) alone.push(i); // World 2: 8192 threads share the SAME cache. Each has its own tiny 4-line // working set and they run interleaved, warp-style — everyone revisits // their data ten times, just like world 1. const crowd: number[] = []; for (let pass = 0; pass < 10; pass++) for (let t = 0; t < 8192; t++) for (let i = 0; i < 4; i++) crowd.push(t * 4 + i); console.log(`lone thread, 256-line set, 10 passes: hit rate ${(100 * hitRate(alone)).toFixed(1)}%`); console.log(`8192 threads x 4 lines, 10 passes: hit rate ${(100 * hitRate(crowd)).toFixed(1)}%`);

Identical reuse in the program — every line is touched ten times in both worlds — yet the crowd gets almost nothing: 8192 × 4 = 32,768 distinct lines cycle through 1,024 slots, and every revisit finds its line evicted. Shrink the crowd to 128 threads (512 lines total) and the hit rate snaps back. That is the arithmetic of 20 bytes per thread, acted out.

Recent flagship GPUs ship tens of megabytes of L2 (or an extra "infinity" level of last-level cache) — a strange investment in a structure this lesson keeps calling tiny. The economics are on the bandwidth side: every byte served from L2 is a byte GDDR/HBM doesn't ship, and DRAM bandwidth is the most expensive commodity on the board — wide buses, exotic packaging, stacked dies, watts. Growing the L2 is cheaper than growing the memory bus, so vendors buy effective bandwidth with SRAM: a 50% L2 hit rate doubles what the DRAM system appears to deliver. The cache grew, but its job description didn't — it is a bigger filter, not a latency machine.

A classic wasted afternoon: restructuring a kernel so each thread revisits its own data "while it's hot in L1" — CPU-style temporal-locality tuning. But your thread's share of the L1 is ~64 bytes; by the time the warp scheduler has cycled through a few dozen other warps, "your" lines are long gone. On a GPU you engineer reuse in the two memories you actually control: registers (keep accumulators and loop-carried values in per-thread variables — 128 B of guaranteed residence) and shared memory (stage tiles explicitly, block-wide). Leave the caches exactly one job: make your access patterns coalesced and let the L1/L2 quietly absorb the spillover and cross-thread sharing. Reuse you can't name and place is reuse you don't have.