Running a Kernel End to End

Four lessons of parts; now the machine. This is the payoff lesson — the one where an ISA on paper, a lockstep core, a dealing dispatcher and a bottlenecked memory controller click together and PrimerGPU breathes: a kernel goes in one end and eight correct sums come out the other, with every one of the 93 cycles in between accounted for. We will watch the whole life of a launch — dispatch, lockstep fetch, per-lane index math, coalesced loads queueing, the honest stalls, stores, RET, done, kernel-complete — then interrogate the run like engineers: where did the cycles go? and what happens if we take pieces away? Three configurations, one truth about GPUs, and a self-check in the proper testbench tradition to prove none of it is wishful thinking.

The complete machine

Every box on this diagram is something you built: the contract it all serves, the cores from lesson two, the dispatcher from lesson three, the memory system from lesson four. The simulator below contains all of it in about 150 lines — dispatcher, both cores, queues, arbitration, coalescer — still the same two-phase settle-then-snap loop we started with back in the RTL module, just with a grander State.

THE run

The launch: vector add over 8 elements as 2 blocks of 4 threads, on 2 cores with the coalescer on. One line per cycle — each core's FSM state, each channel's countdown, and annotations when something happens. Then the accounting, and then the same kernel with pieces removed:

// ── The whole PrimerGPU: dispatcher + 2 SIMT cores + memory controller + coalescer. ── // Kernel: vector add, 8 elements as 2 blocks of 4 threads. a@0, b@8, c@16. const PROG = [0x50de, 0x300f, 0x9100, 0x9208, 0x9310, 0x3410, 0x7440, 0x3520, 0x7550, 0x3645, 0x3730, 0x8076, 0xf000]; const NUM_BLOCKS = 2, LATENCY = 4; const chanOf = (a: number) => Math.floor(a / 4) % 2; type Fsm = "FETCH" | "DECODE" | "REQUEST" | "WAIT" | "EXECUTE" | "UPDATE" | "IDLE" | "DONE"; const STEP: Record<string, Fsm> = { FETCH: "DECODE", DECODE: "REQUEST", REQUEST: "WAIT", WAIT: "EXECUTE", EXECUTE: "UPDATE", UPDATE: "FETCH", }; interface Lane { regs: number[]; data: number } interface Core { st: Fsm; pc: number; ir: number; pending: number; blockIdx: number; lanes: Lane[] } interface Req { kind: "load" | "store"; addr: number; core: number; lane: number; data: number[] } interface Chan { queue: Req[]; cur: Req | null; left: number } interface M { cores: Core[]; chans: Chan[]; mem: number[]; nextBlock: number; doneBlocks: number; txns: number } const idle = (): Core => ({ st: "IDLE", pc: 0, ir: 0, pending: 0, blockIdx: -1, lanes: [] }); const start = (b: number): Core => ({ st: "FETCH", pc: 0, ir: 0, pending: 0, blockIdx: b, lanes: [0, 1, 2, 3].map((t) => { const regs = new Array(16).fill(0); regs[13] = b; regs[14] = 4; regs[15] = t; return { regs, data: 0 }; }), }); function coreNext(s: Core, ci: number, out: Req[], coalesce: boolean): Core { if (s.st === "IDLE" || s.st === "DONE") return s; const n: Core = { ...s, st: STEP[s.st], lanes: s.lanes.map((l) => ({ ...l, regs: [...l.regs] })) }; const w = s.ir, op = w >> 12, rd = (w >> 8) & 15, rs = (w >> 4) & 15, rt = w & 15, imm = w & 255; if (s.st === "FETCH") n.ir = PROG[s.pc]; if (s.st === "REQUEST" && (op === 7 || op === 8)) { const kind = op === 7 ? "load" as const : "store" as const; const addrs = s.lanes.map((l) => l.regs[rs]); if (coalesce && addrs.every((a, t) => a === addrs[0] + t)) { out.push({ kind, addr: addrs[0], core: ci, lane: -1, data: s.lanes.map((l) => l.regs[rt]) }); n.pending = 1; } else { s.lanes.forEach((l, t) => out.push({ kind, addr: addrs[t], core: ci, lane: t, data: [l.regs[rt]] })); n.pending = 4; } } if (s.st === "WAIT" && s.pending > 0) n.st = "WAIT"; if (s.st === "EXECUTE") n.lanes.forEach((l) => { const a = l.regs[rs], b = l.regs[rt]; l.data = op === 3 ? a + b : op === 4 ? a - b : op === 5 ? a * b : op === 6 ? Math.floor(a / b) : op === 9 ? imm : l.data; }); if (s.st === "UPDATE") { if ((op >= 3 && op <= 7) || op === 9) n.lanes.forEach((l) => { if (rd < 13) l.regs[rd] = l.data; }); if (op === 15) n.st = "DONE"; n.pc = s.pc + 1; } return n; } function machineNext(m: M, rr: number, coalesce: boolean, note: (s: string) => void): M { const mem = [...m.mem]; const out: Req[] = []; const cores = m.cores.map((c, i) => coreNext(c, i, out, coalesce)); const chans = m.chans.map((ch) => ({ ...ch, queue: [...ch.queue] })); let { nextBlock, doneBlocks, txns } = m; // memory channels: grind, deliver by requestor ID chans.forEach((ch, k) => { if (ch.cur) { ch.left--; if (ch.left === 0) { const r = ch.cur, c = cores[r.core]; if (r.kind === "load") { if (r.lane < 0) c.lanes.forEach((l, t) => { l.data = mem[r.addr + t]; }); else c.lanes[r.lane].data = mem[r.addr]; } else if (r.lane < 0) r.data.forEach((d, t) => { mem[r.addr + t] = d; }); else mem[r.addr] = r.data[0]; c.pending--; note(`ch${k} answers core ${r.core}${r.lane < 0 ? " (all 4 lanes)" : " lane " + r.lane}`); ch.cur = null; } } }); // arbitration: rotating requestor priority into the channel queues const rid = (r: Req) => r.core * 4 + Math.max(r.lane, 0); out.sort((x, y) => ((rid(x) - rr + 8) % 8) - ((rid(y) - rr + 8) % 8)); out.forEach((r) => { chans[chanOf(r.addr)].queue.push(r); txns++; }); chans.forEach((ch) => { if (!ch.cur && ch.queue.length > 0) { ch.cur = ch.queue.shift()!; ch.left = LATENCY; } }); // dispatcher: collect done, grant one block per cycle to the lowest free core m.cores.forEach((c, i) => { if (c.st === "DONE") { doneBlocks++; cores[i] = idle(); note(`core ${i} done (block ${c.blockIdx}) — ${doneBlocks}/${NUM_BLOCKS}`); } }); const free = cores.findIndex((c) => c.st === "IDLE"); if (nextBlock < NUM_BLOCKS && free >= 0) { cores[free] = start(nextBlock); note(`dispatch block ${nextBlock} -> core ${free}`); nextBlock++; } return { cores, chans, mem, nextBlock, doneBlocks, txns }; } function run(numCores: number, coalesce: boolean, verbose: boolean) { const mem = new Array(24).fill(0); for (let i = 0; i < 8; i++) { mem[i] = 10 * (i + 1); mem[8 + i] = i + 1; } let m: M = { cores: Array.from({ length: numCores }, idle), chans: [{ queue: [], cur: null, left: 0 }, { queue: [], cur: null, left: 0 }], mem, nextBlock: 0, doneBlocks: 0, txns: 0, }; const budget: Record<string, number> = {}; let cycle = 0, waitCycles = 0; while (m.doneBlocks < NUM_BLOCKS) { const notes: string[] = []; const snap = m; m = machineNext(m, cycle % 8, coalesce, (s) => notes.push(s)); snap.cores.forEach((c) => { budget[c.st] = (budget[c.st] ?? 0) + 1; if (c.st === "WAIT") waitCycles++; }); if (verbose) { const cs = snap.cores.map((c, i) => `core${i} ${(c.st === "FETCH" ? "FETCH p" + c.pc : c.st).padEnd(8)}`).join(" "); const ch = snap.chans.map((c, k) => `ch${k} ${c.cur ? "busy" + c.left : "- "}${c.queue.length > 0 ? "+q" + c.queue.length : " "}`).join(" "); console.log(`c${String(cycle).padStart(2, "0")} ${cs} ${ch} ${notes.join("; ")}`); } cycle++; } // self-check: the RTL's memory must equal a plain software reference (the testbench discipline) const ref = new Array(24).fill(0); for (let i = 0; i < 8; i++) { ref[i] = 10 * (i + 1); ref[8 + i] = i + 1; ref[16 + i] = ref[i] + ref[8 + i]; } const pass = m.mem.every((x, i) => x === ref[i]); return { cycle, txns: m.txns, budget, waitCycles, pass, c: m.mem.slice(16) }; } // ── THE run: the full machine, watched cycle by cycle. ── console.log("PrimerGPU end to end: 2 cores, coalescer ON, 2 blocks of 4 threads"); const A = run(2, true, true); console.log(`\nkernel complete at cycle ${A.cycle}; c = [${A.c.join(", ")}]; self-check ${A.pass ? "PASS" : "FAIL"}`); console.log("\nwhere the core-cycles went (both cores, config A):"); const total = Object.values(A.budget).reduce((a, b) => a + b, 0); (["FETCH", "DECODE", "REQUEST", "WAIT", "EXECUTE", "UPDATE", "IDLE"] as const).forEach((st) => { const n = A.budget[st] ?? 0; console.log(` ${st.padEnd(7)} ${String(n).padStart(4)} cycles ${String(Math.round((100 * n) / total)).padStart(3)}% ${"#".repeat(Math.round(n / 4))}`); }); console.log("\nthree configurations, same kernel, same data:"); const B = run(1, true, false); const C = run(2, false, false); console.log(` A: 2 cores, coalescer ON : ${A.cycle} cycles, ${A.txns} transactions (self-check ${A.pass ? "PASS" : "FAIL"})`); console.log(` B: 1 core, coalescer ON : ${B.cycle} cycles, ${B.txns} transactions (self-check ${B.pass ? "PASS" : "FAIL"})`); console.log(` C: 2 cores, coalescer OFF: ${C.cycle} cycles, ${C.txns} transactions (self-check ${C.pass ? "PASS" : "FAIL"})`); console.log(` parallelism (B/A): ${(B.cycle / A.cycle).toFixed(2)}x coalescing (C/A): ${(C.cycle / A.cycle).toFixed(2)}x`);

Now walk the trace like a guided tour — every phase of a kernel's life is in there:

Where the cycles went

The budget table is the honest heart of this lesson:

StateCore-cyclesShareWhat it is
FETCH2614%reading instruction words
DECODE2614%unpacking fields
REQUEST2614%computing addresses, issuing requests
WAIT5027%stalled on the memory system
EXECUTE2614%the ALUs actually computing
UPDATE2614%committing registers and the PC
IDLE42%launch and drain ramps

WAIT is the largest line — nearly double any other state — and that is with the coalescer doing its best; switch it off (config C) and waiting swells to roughly half of all core time. Meanwhile EXECUTE, the only state where mathematics actually happens, gets one cycle in seven. Our toy, built honestly, has independently discovered the central truth of GPU design: a GPU is a machine for surviving memory latency, and everything real GPUs add — warps in flight, caches, deeper coalescing, occupancy machinery — is aimed squarely at that 27%.

Three machines, one kernel

The comparison table quantifies the two big architectural ideas on our own machine, not by appeal to authority:

ConfigCoresCoalescerCyclesTransactionsvs. A
A2on936
B1on18361.97× slower
C2off129241.39× slower

Halve the cores and the kernel takes 1.97× longer — block-level dispatch converting hardware directly into speed, the scalability contract paying out at N{=}2. Keep both cores but scatter the memory traffic into per-lane transactions and you lose 1.39× while moving the same 24 words. And in every configuration, the self-check passes: the final data memory is compared, word for word, against a reference computed by a plain software loop — the testbench discipline from the verification module, in miniature. A trace that looks right is an anecdote; a machine-checked equality against an independent model is evidence. The crown jewel is not that the trace is pretty — it is that line: self-check PASS.

Pause on what just happened, because it is the whole course in one breath. You wrote text — thirteen assembly lines. A program you wrote (the assembler) turned it into sixteen-bit words. A machine you specified at the register-transfer level — FSMs, latches, arbitration, all things that synthesize to gates — executed those words, and a testbench you understand proved the answer right. That is the same loop a thousand-engineer GPU team runs, and the same loop this course's earlier capstone ran for a CPU; the difference is only scale and the number of zeros on the budget. There is a marvellous circularity available here too: our simulator ran on your computer's processor — silicon that began life as somebody's RTL, simulated on some earlier processor, all the way back down a chain of machines building machines that bottoms out in the hand-drawn masks of the 1970s. You are now, in a small but genuine way, part of that chain.

It would be easy to read the budget table as an indictment — our cores twiddle in WAIT over a quarter of the time; surely a better design would fix that? Look closer: the fix is not cleverer circuits, it is more threads. Our machine runs 8 threads, so when both blocks stall on loads there is nothing else to switch to — the WAIT shows up as naked idle time. A real SM keeps dozens of warps resident precisely so that another warp's arithmetic can fill every one of those cycles: same latency, hidden behind other work. Real GPUs run hundreds of thousands of threads not because the workload demands that many answers at once, but because that is how many are needed to bury the memory stalls our toy displays nakedly. In other words: PrimerGPU's inefficiency is the most honest thing about it — it exhibits the disease (latency) with the medicine (massive multithreading) removed, which is exactly why the medicine exists. The next lesson makes that upgrade path precise.