The Warp Scheduler

Strip away everything else and a GPU's magic trick is performed by one small circuit, four times per SM: the warp scheduler. Its job description fits in a sentence — every cycle, look at your dozen resident warps, pick one that is ready, and issue its next instruction — and that sentence is the entire mechanism of latency hiding. While one warp waits three hundred cycles for DRAM, the scheduler simply keeps dealing instructions from the others. No prediction, no out-of-order window, no cleverness about any single thread: just always having someone else to run. This lesson is the heartbeat of the SM, slowed down until you can see every tick.

Ready or not: the scoreboard

"Pick a ready warp" does all the work in that sentence, so what makes a warp ready? The scheduler keeps a scoreboard — a little table of per-warp dependency state. A warp is not ready when:

its next instruction needs a value from a load that hasn't returned yet (the scoreboard marks the destination register "in flight" at issue and clears it when the data arrives — possibly hundreds of cycles later);
it is waiting on a slow arithmetic result still in the pipeline;
the execution unit it needs is busy — the FP32 lanes, an SFU, a load/store port — or it's parked at a barrier waiting for other warps.

Everyone else is fair game. Among the ready warps, real schedulers use simple policies: loose round-robin (rotate through ready warps, keeping everyone moving evenly) or greedy-then-oldest (keep issuing from the same warp until it stalls — friendlier to its cache lines — then fall back to the oldest ready warp). Nothing fancier fits: the decision must be made every single cycle, so it has to be almost free.

each cycle, each scheduler picks one ready warp and issues its next instruction to an execution unit;
scoreboarding tracks per-warp readiness: outstanding loads, pipeline results, busy units;
switching warps costs zero cycles, because every resident warp's state already lives in the register file;
latency is hidden by interleaving: enough ready warps ⇒ the issue slot never goes idle.

The zero-cost context switch

Here is the trick that separates a GPU from every CPU threading scheme. When an operating system switches threads, it saves one thread's registers to memory and restores another's — thousands of cycles. When a CPU core with simultaneous multithreading switches between its two hardware threads, it avoids the save/restore by keeping both threads' registers on-chip — the GPU's cousin, at width two. The SM takes that idea to its logical extreme: all 48–64 resident warps' registers live permanently in the register file. Nothing is ever saved or restored. "Switching" from warp 3 to warp 7 means the scheduler's mux selects a different row of state — it is literally just issuing a different warp's instruction next cycle. Free.

That's why the previous lesson made such a fuss about the SM housing its warps. The multitude isn't kept anywhere convenient — it is wired directly into the machine, so that the scheduler can flick between threads at cycle granularity. A CPU hides memory latency with caches and out-of-order magic; the SM hides it by having the next warp's instruction ready before the stalled warp has even finished stalling.

Worked example: scheduling four warps by hand

Give every warp the same little program, over and over: one compute instruction C, then a load M whose result the next C needs. A load's data returns 8 cycles after issue; the scheduler issues one instruction per cycle, loose round-robin. With one warp the story is grim: it issues C, issues M, then the scoreboard blocks it for 8 cycles — two busy cycles out of every nine, \approx 22\% utilisation. Now watch four warps:

Cycles 0–3: everyone's C. Cycles 4–7: everyone's M. Then the bubble — cycles 8–11 have no ready warp, because warp 0's load (issued at cycle 4) only returns at cycle 12. Steady state: 8 issues every 12 cycles, 67\% utilisation. With eight warps the bubble vanishes entirely — by the time warp 7 has issued its load, warp 0's data is back: 100\%. Same code, same latency; the only thing that changed is how much other work the scheduler could deal while the memory system did its slow thing. That is "latency hiding" at instruction level: nothing is faster, it is merely interleaved.

The scheduler in the simulator

Our two-phase pattern — settle, then snap — fits the scheduler perfectly: settling is scanning the scoreboard for a ready warp; snapping is committing the issue and updating that warp's state. Here is the whole thing, running the worked example for 1, 4 and 8 warps:

// A warp alternates: compute C (ready next cycle), load M (dependent instr ready after LATENCY). interface Warp { id: number; nextIsLoad: boolean; readyAt: number } const LATENCY = 8; function simulate(numWarps: number, cycles: number, trace: boolean): number { const warps: Warp[] = []; for (let id = 0; id < numWarps; id++) warps.push({ id, nextIsLoad: false, readyAt: 0 }); let rr = 0, issued = 0, line = ""; for (let c = 0; c < cycles; c++) { // settle: scan round-robin for the first READY warp (the scoreboard check) let pick = -1; for (let k = 0; k < numWarps; k++) { const w = warps[(rr + k) % numWarps]; if (w.readyAt <= c) { pick = w.id; break; } } // snap: commit the issue and update that warp's scoreboard entry if (pick >= 0) { const w = warps[pick]; line += w.nextIsLoad ? `M${w.id} ` : `C${w.id} `; w.readyAt = w.nextIsLoad ? c + LATENCY : c + 1; w.nextIsLoad = !w.nextIsLoad; rr = (pick + 1) % numWarps; issued++; } else line += "--- "; // no ready warp: the issue slot goes idle } if (trace) console.log(` cycles 0..${cycles - 1}: ${line}`); return issued / cycles; } console.log("4 warps, first 12 cycles (Cn/Mn = warp n issues, --- = stall):"); simulate(4, 12, true); console.log(""); for (const n of [1, 4, 8]) { const u = simulate(n, 216, false); console.log(`${n} warp(s): utilisation ${(u * 100).toFixed(0)}%`); }

Change LATENCY to 16 and watch 8 warps stop being enough — the required warp count scales with the latency to be covered. That relationship gets its own capstone lesson at the end of this module, under the name occupancy.

Seymour Cray did — in 1964, for the CDC 6600, the first machine anyone called a supercomputer. Its "scoreboard" was a unit that tracked which registers were waiting on which functional units so the machine could keep issuing independent instructions instead of stalling — dependency tracking in hardware, sixty years before your GPU. The CDC 6600 used it to keep one instruction stream busy across ten functional units; an SM uses the same bookkeeping to juggle dozens of warps across its lanes. Computer architecture recycles its best ideas: the scoreboard, born to extract parallelism within a thread, was reborn as the referee between threads.

If 8 warps beat 4, surely 48 beat 8? No — once the latency is covered, utilisation is already 100% and extra warps hide nothing more. They aren't free, either: every additional resident warp takes its slice of the register file and dilutes each warp's share of the L1 and shared memory, which can make the memory behaviour worse. Resident-warp count (occupancy) is a means — the goal is hidden latency, and past the covering threshold you are paying real resources for zero benefit. Keep this instinct warm: the final lesson of this module turns it into an engineering method.