Occupancy and Latency Hiding
Every lesson in this module has been circling one question. The
scheduler
hides latency if it has enough ready warps; the
caches
don't hide it at all; registers and
shared
memory are both wonderful and both, we kept hinting, "a currency". This capstone
cashes the hints. Occupancy — the fraction of an SM's warp slots actually filled —
is the number that connects your kernel's resource appetite to the scheduler's ability to hide
latency, and it is governed by an equation humble enough to compute on paper and consequential
enough that every GPU vendor ships a calculator for it. By the end of this lesson you will
be that calculator — and, more importantly, know when to ignore it.
One SM, three budgets
An SM has a fixed stock of the things a resident thread block consumes. A block runs only
if all of its appetites fit simultaneously — so residency is a \min
over independent budgets:
- \text{occupancy} = \frac{\text{resident warps}}{\text{max warp slots}}
- resident warps = \min over three budgets:
registers (register file ÷ registers per thread),
shared memory (per-SM pool ÷ per-block request), and
slots (hard caps on warps and blocks per SM);
- whichever budget produces the minimum is the binding constraint — spending
less of it is the only change that raises occupancy;
- occupancy is a means: enough warps to cover latency — not a score to
maximise.
Worked example, all three budgets. Take a representative SM — 65,536 registers,
96 KB of shared memory, 48 warp slots, 16 block slots — and a
kernel launched in blocks of 256 threads (8 warps) using 64 registers per
thread and 12 KB of shared memory per block:
| Budget | Arithmetic | Allows |
|---|
| Registers | 65,536 ÷ (64 × 32 per warp) = 32 warps | 32 warps |
| Shared memory | ⌊96 ÷ 12⌋ = 8 blocks × 8 warps | 64 warps |
| Slots | hard cap | 48 warps |
Residency = \min(32, 64, 48) = 32 warps; occupancy
= 32/48 \approx 67\%, and the binding budget is
registers. That last fact is the actionable one: shrinking shared memory would change
nothing, but coaxing the compiler down from 64 to 42 registers per thread would lift residency to
48 warps. The tuning loop is exactly this, iterated: profile → identify the binding budget →
trade some of it → re-measure.
How many warps are "enough"? Little's law
The Iron
Law taught you to decompose time; its queueing-theory cousin,
Little's law, tells you how much concurrency a latency demands:
\text{bytes in flight} = \text{bandwidth} \times \text{latency}.
To sustain 2000 GB/s of DRAM bandwidth against a
400 ns memory latency, the chip must have
2000 \times 400 \approx 800{,}000 bytes — 800 KB —
of loads in flight at every instant. Split over 100 SMs, each SM owes the memory system
8 KB of outstanding requests; at 128 bytes per coalesced warp access, that is
64 requests in flight per SM. If a typical warp keeps two loads outstanding, you
need 32 resident warps busy issuing — right around our 67%-occupancy example.
This is the honest job description of occupancy: supply the concurrency Little's law
demands. The scheduler's zero-cost switching provides the mechanism; occupancy provides the
raw material.
Two corollaries follow immediately. Memory-bound kernels feel occupancy keenly —
halve the warps and you halve the in-flight bytes, and the DRAM pipe runs half-empty.
Compute-bound kernels barely notice: if the ALUs are already saturated at 50%
occupancy, more warps add nothing but cache pressure. And warps are not the only source of
in-flight requests: a single warp whose loop is unrolled can have four or eight
independent loads outstanding at once — instruction-level parallelism substituting for
occupancy. Some of the fastest kernels ever written run at 25% occupancy with huge
register budgets, each warp streaming like a freight train.
The cliffs
Occupancy is a staircase, not a slope: warps are whole, blocks are whole, so residency falls in
cliffs as per-thread appetite grows. The dashed curve is the register budget alone; the solid
curve is the actual \min of all three. Drag the shared-memory slider
and watch the binding constraint switch — the flat ceiling that appears at high
shared-memory use is the shared budget taking over from registers:
Note the plateaus: between two cliffs, spending more registers is free. If the compiler
is using 50 registers, occupancy is identical at 60 — a fact that matters in the Watch out below.
The occupancy calculator, executable
The vendor tool that every GPU programmer eventually meets is a few lines of
\min and \lfloor\cdot\rfloor. Here it is,
sweeping register counts for our worked-example kernel and naming the binding budget at each step:
// The occupancy calculator: three budgets, one min.
const SM = { regs: 65536, smemKB: 96, warpSlots: 48, blockSlots: 16 };
function occupancy(regsPerThread: number, smemPerBlockKB: number, threadsPerBlock: number) {
const wpb = threadsPerBlock / 32; // warps per block
const byRegs = Math.floor(SM.regs / (regsPerThread * 32));
const bySmem = Math.floor(SM.smemKB / smemPerBlockKB) * wpb;
const bySlots = Math.min(SM.warpSlots, SM.blockSlots * wpb);
const budgets: [string, number][] = [
["registers", byRegs], ["shared mem", bySmem], ["slots", bySlots],
];
const cap = Math.min(byRegs, bySmem, bySlots);
const warps = Math.floor(cap / wpb) * wpb; // blocks are all-or-nothing
const binding = budgets.find(([, w]) => w === cap)![0];
return { warps, pct: Math.round((100 * warps) / SM.warpSlots), binding };
}
console.log("256-thread blocks, 12 KB shared per block:");
console.log("regs/thread warps occupancy binding budget");
for (let r = 32; r <= 128; r += 16) {
const o = occupancy(r, 12, 256);
console.log(` ${String(r).padStart(3)} ${String(o.warps).padStart(2)} ${String(o.pct).padStart(3)}% ${o.binding}`);
}
console.log("");
console.log("same kernel, but 24 KB shared per block:");
for (const r of [32, 64]) {
const o = occupancy(r, 24, 256);
console.log(` ${String(r).padStart(3)} ${String(o.warps).padStart(2)} ${String(o.pct).padStart(3)}% ${o.binding}`);
}
Read the sweep like a tuning session: the 48-register row and the 32-register row land on
different budgets, and once shared memory doubles, the register knob stops mattering entirely —
turning the wrong knob is the classic wasted optimisation, and the calculator exists to
point at the right one.
For years the official "CUDA Occupancy Calculator" was, literally, an Excel spreadsheet — a
yellow-and-green grid shipped with the CUDA toolkit where you typed three numbers (registers per
thread, shared memory per block, threads per block) and a bar chart told you your occupancy and
which budget bound it. It became the shared ritual of an entire field: conference talks showed
screenshots of it; forum advice began "open the calculator". Its logic — the same dozen lines you
just ran — now lives inside the profiler and even inside the runtime API, but the spreadsheet
deserves its footnote in history: it taught a generation of programmers that GPU performance
tuning starts not with cleverness but with an inventory.
Occupancy is the rare metric that is dangerous to maximise. Past the point where latency
is covered, extra warps hide nothing more — the scheduler's issue slot was already never idle —
but they keep costing: every additional resident warp dilutes each warp's share of the
register file, L1 and shared memory, which can force the compiler to spill registers to
memory and turn a compute-bound kernel into a memory-bound one. Chasing 100% typically means
starving each thread down to ~42 registers and forgoing the unrolling that gives you ILP — trading
away per-warp strength for warm-body count you didn't need. The professionals' rule:
occupancy must be sufficient, not maximal — enough warps to satisfy
Little's law for your kernel's actual traffic, and beyond that, spend the budgets on making each
warp faster. Ship the fastest kernel, not the fullest SM.
The module, in one breath
You can now tell the whole story of this module as one causal chain. The SM
houses dozens of warps; the scheduler interleaves them to hide latency;
divergence and un-coalesced access waste the lanes and the bandwidth the interleaving is
supposed to feed; shared memory multiplies arithmetic intensity so less bandwidth is
needed; the caches filter what traffic remains; and occupancy — three budgets, one
\min, Little's law — decides whether the scheduler has enough warps to
make the whole trick work. Everything else in GPU performance engineering is these six sentences,
applied.