Measuring GPU Performance

The profiler delivers its verdict: your kernel is achieving 4% of the GPU's peak flops. Four percent! Of the machine you so carefully launched work onto! Before you refactor anything, this lesson wants to install one reflex: 4% of which roof? A GPU has two ceilings — arithmetic and memory bandwidth — and a kernel can only ever be judged against the one that applies to it. The roofline model gave us the theory; today we point it at a real GPU, add the second-most-important number on the chip (occupancy), and assemble the short list of measurements that turns "it's slow" into a diagnosis.

The ridge of a modern GPU

Recall the machinery: a kernel's arithmetic intensity I is flops performed per byte moved from DRAM, and attainable performance is P = \min(\pi,\; I\beta). The ridge point I^\star = \pi/\beta separates memory-bound from compute-bound. Now put GPU-sized numbers in. A respectable accelerator: \pi = 20 Tflop/s, \beta = 1 TB/s:

I^\star = \frac{20\,000\ \text{Gflop/s}}{1\,000\ \text{GB/s}} = 20\ \text{flop/byte}.

Twenty flops per byte touched just to break even with the ALUs. Now audit some kernels:

Kernel	Intensity I	Verdict on a ridge-20 machine
vector add c_i = a_i + b_i	1 flop / 12 B ≈ 0.08	memory-bound, 250× below the ridge — roof is 83 Gflop/s, 0.4% of peak
stencils, elementwise ML ops	0.5 – 2	memory-bound — this is most real code
tiled matrix multiply	hundreds (grows with tile size)	compute-bound — the rare kernel that can touch peak flops

The uncomfortable truth of GPU performance work is in that middle row: most kernels live far left of the ridge, where the memory roof — not the shiny Tflops number — is the law.

Slide along the roof

Practitioners draw the roofline on log–log axes so that five orders of magnitude of intensity fit on one chart: the memory roof becomes a 45° line, the compute roof stays flat, and they meet at the ridge. Drag the slider — it is your kernel's intensity — and watch the marker ride up the memory roof, hit the ridge at I = 20 (\log_{10} I \approx 1.3), and go no higher. The anchors mark vector add (≈0.08), a stencil (≈3), the ridge, and a well-tiled matmul (≈130):

Everything left of the ridge can be read as a percentage: at I = 0.08 the roof sits a factor of 250 below peak — so full marks for a vector add is 0.4% of peak flops.

Occupancy: are there enough warps to hide the wait?

The roofline says what the hardware could deliver at your intensity; whether you get it depends on keeping the machine busy while memory crawls. A DRAM access costs hundreds of cycles, and the GPU's whole survival strategy is to have other warps ready to issue during the wait. The first-order health metric for that strategy is occupancy:

\text{occupancy} \;=\; \frac{\text{resident warps per SM}}{\text{maximum resident warps per SM}}.

Why would it ever be below 1? Because residency is rationed by the SM's fixed budgets: the register file and shared memory are carved up among resident blocks. A kernel that uses many registers per thread, or a big slab of shared memory per block, simply fits fewer warps. Worked example: an SM holds at most 64 warps and 65,536 registers; a kernel using 40 registers per thread needs 40 \times 32 = 1280 registers per warp, so at most \lfloor 65536/1280 \rfloor = 51 warps fit — rounded down to block granularity, say 48 resident: occupancy 48/64 = 75\%. Low occupancy doesn't always hurt — a kernel with lots of independent work per thread can hide latency with fewer warps — but when a memory-bound kernel misses even its memory roof, occupancy is the first gauge to read. The gpu-compute module finishes this story properly.

The measurement kit, and the diagnosis tree

One more distinction: effective versus peak bandwidth. Peak is the datasheet; effective is useful bytes your kernel actually moved per second. They diverge when warps touch scattered addresses, dragging whole memory transactions across the bus for a few useful bytes — the coalescing story, previewed here and told in full in its own lesson. With that, the kit is complete. Measure three numbers, then walk the tree:

Measurements say…	Diagnosis	Your move
achieved bandwidth ≈ peak, flops% low, I < I^\star	memory-bound, at its roof — healthy	raise intensity (tiling, reuse, fusion) or accept victory
bandwidth far below peak, still memory-bound	wasted bytes on the bus	fix access patterns — coalescing, layout, smaller types
occupancy low, both other numbers low	latency not hidden; the SM idles between dependent ops	cut registers / shared memory per thread, resize blocks
flops ≈ peak	compute-bound, at the flat roof	algorithmic change, lower precision, or a bigger GPU

arithmetic intensity I (flop/byte) decides the applicable roof: P = \min(\pi, I\beta), ridge at I^\star = \pi/\beta;
occupancy = resident warps ÷ maximum — the latency-hiding budget, capped by per-thread registers and per-block shared memory;
effective bandwidth counts useful bytes; scattered access burns the gap to peak;
measure achieved bandwidth %, achieved flops %, occupancy — then diagnose against the roof that applies, never against peak flops alone.

The calculator

The whole methodology fits in a dozen lines. Feed it a machine and a kernel's flops and bytes, and it names the wall and the ceiling:

// Roofline calculator: which wall does each kernel hit, and how fast can it go? const peak = 20000; // Gflop/s (20 Tflop/s) const bw = 1000; // GB/s (1 TB/s) const ridge = peak / bw; console.log(`machine: ${peak} Gflop/s, ${bw} GB/s -> ridge = ${ridge} flop/byte\n`); const kernels = [ { name: "vector add c=a+b ", flops: 1, bytes: 12 }, // 2 reads + 1 write per flop { name: "7-point stencil ", flops: 13, bytes: 8 }, // good reuse from shared memory { name: "tiled matmul (128)", flops: 128, bytes: 1 }, // ~tile-width flops per byte ]; for (const k of kernels) { const I = k.flops / k.bytes; const roof = Math.min(peak, I * bw); const bound = I < ridge ? "MEMORY-bound " : "COMPUTE-bound"; const pct = ((roof / peak) * 100).toFixed(1).padStart(5); console.log(`${k.name} I=${I.toFixed(2).padStart(7)} ${bound} roof=${roof.toFixed(0).padStart(5)} Gflop/s = ${pct}% of peak flops`); } const vaddRoof = Math.min(peak, (1 / 12) * bw); console.log(`\nA vector add at ${((vaddRoof / peak) * 100).toFixed(1)}% of peak flops is the hardware`); console.log("working PERFECTLY - that IS the roof at intensity 0.08.");

Try your own kernel: count its flops and its DRAM bytes per element, add a line, rerun. Then try doubling bw (a next-generation memory system) — watch every memory-bound roof double while the matmul doesn't budge.

The number on the box assumes every FMA unit on the chip issues a fused multiply-add every cycle, forever — no loads, no stores, no dependencies, no divergence, and a kernel that is 100% multiply-adds (an FMA counts as two flops, which conveniently doubles the figure). Peak is not a promise; it is the product of clock frequency and unit count — a statement about what the ALUs cost, not what your program gets. The roofline is the honest translation layer: it converts the box number into your number by asking the one question marketing never does — how many bytes must move per flop? That is also why vendors increasingly quote peak memory bandwidth in the same breath, and why the ratio of the two numbers (the ridge) tells you more about a GPU's personality than either alone. Two chips with identical Tflops but bandwidths a factor of two apart are, for most real kernels, a factor of two apart.

A dashboard shows two kernels: kernel A at 61% of peak flops, kernel B at 4%. Which needs the engineer? Possibly A. If B is a memory-bound kernel with I = 0.8 on a ridge-20 machine, its roof is 0.8/20 = 4\% of peak flops — B is delivering exactly what physics allows: the hardware is working perfectly, and the profiler will confirm it by showing memory bandwidth pinned at ~100%. Meanwhile A, if it is a matmul that should be compute-bound near 90%, is leaving a third of the machine idle — perhaps to poor occupancy or a shared-memory bank conflict. The rule: a utilisation percentage means nothing until you know which roof applies. Judge every kernel against \min(\pi, I\beta) at its own intensity — never against bare peak — or you will spend weeks "optimising" kernels that were already touching their ceiling, while the genuinely broken ones sail past code review with a big green number.