Memory Controllers in Hardware

Every master we've put on the fabric — CPU, DMA engines, GPU — ultimately funnels into one place: the DRAM. And DRAM, as you already know, is a diva. It is organised as banks of rows of columns; before you can read a byte you must activate its row (copy a whole row into the bank's row buffer, waiting t_{RCD}), then issue the column access (waiting t_{CAS}); to touch a different row you must first precharge the bank (t_{RP}) and start over. A row hit costs one cheap column access; a row conflict costs the whole precharge–activate–access ritual — several times more. And every few microseconds the whole chip demands a refresh or it forgets your data.

The prerequisite taught you the diva's personality. This lesson builds her handler: the memory controller, the hardware block that accepts a chaotic stream of requests from the fabric and turns it into a legal, ruthlessly optimised command schedule. It is one of the most consequential blocks on the die — the difference between a naive controller and a good one is often 2× on delivered bandwidth, for free, forever.

Anatomy: queues, counters, and a scheduler

Open the hood and the controller has three organs:

Per-bank request queues. Arriving reads and writes are decoded (address → bank, row, column) and parked in a queue per bank. Why per bank? Because banks are independent little machines: while bank 0 is busy transferring data, bank 2 can be activating a row. Separate queues let the scheduler see — and exploit — that bank-level parallelism.
The timing FSM. Every JEDEC parameter becomes a hardware counter: on ACT, a per-bank counter loads t_{RCD} and counts down; the FSM simply refuses to issue a column command for that bank until it hits zero. Another counter enforces t_{RP} after PRE, another t_{CAS} from column command to data, others the dozen further constraints. Correctness by construction: no command can be emitted while any governing counter is non-zero.
The scheduler. Each cycle, it looks across all the bank queues and asks: of every command that is legal right now, which one do I send? This tiny combinational decision is where the performance lives.

The scheduler: FR-FCFS, and the page-policy bet

The classic answer is FR-FCFS — first-ready, first-come-first-served:

first-ready: prefer requests that are ready to go cheaply — above all, requests that hit a currently open row (a column command now beats an activate-then-column later);
first-come: among equally-ready candidates, oldest wins.

The effect: if the open row has three hits scattered through the queue, the scheduler serves all three before paying the conflict that closes the row — even though a conflicting request arrived earlier. Requests are reordered for row-hit rate and spread across banks for parallelism. It is out-of-order execution, reinvented for memory.

Underneath sits a bet called the page policy. Open-page: after an access, leave the row open, gambling that the next request wants the same row (great for streaming and DMA traffic — hits galore). Closed-page: precharge immediately after each access, gambling that the next request wants a different row, so the t_{RP} is pre-paid and hidden (great for scattered, random traffic). Modern controllers hedge adaptively — track recent hit rate per bank and switch bets.

The schedule, drawn

Step through a few cycles of two banks under FR-FCFS. Compare the width of a hit (RD) with the width of the conflict at the end — that ratio is why the scheduler exists:

Race the schedulers

Here are both schedulers on the same seeded request trace against one bank: plain in-order FCFS versus FR-FCFS. Same requests, same DRAM timings — only the order differs:

// One DRAM bank, two schedulers, one seeded trace. const tCAS = 10, tRCD = 12, tRP = 13; const HIT = tCAS; // row already open const OPEN = tRCD + tCAS; // bank idle: activate, then read const CONFLICT = tRP + tRCD + tCAS; // wrong row open: precharge, activate, read let seed = 2026; const rand = () => { seed = (seed * 1664525 + 1013904223) % 4294967296; return seed / 4294967296; }; // 14 requests over 4 rows, with the burstiness real traffic has. const trace: { id: number; row: number }[] = []; let row = 0; for (let i = 0; i < 14; i++) { if (rand() < 0.45) row = Math.floor(rand() * 4); // sometimes wander to a new row trace.push({ id: i, row }); } console.log("arrival order, by row: " + trace.map((t) => t.row).join(" ")); function run(name: string, pick: (q: { id: number; row: number }[], open: number) => number) { const q = trace.slice(); let open = -1, cycles = 0, hits = 0, misses = 0; const order: number[] = []; while (q.length > 0) { const req = q.splice(pick(q, open), 1)[0]; order.push(req.row); if (req.row === open) { hits++; cycles += HIT; } else { misses++; cycles += open === -1 ? OPEN : CONFLICT; open = req.row; } } console.log(`\n${name}`); console.log(` served order, by row: ${order.join(" ")}`); console.log(` ${hits} row hits, ${misses} activates -> ${cycles} cycles`); return cycles; } const a = run("In-order (FCFS):", () => 0); const b = run("FR-FCFS:", (q, open) => { const hit = q.findIndex((r) => r.row === open); // oldest ready row-hit first... return hit >= 0 ? hit : 0; // ...otherwise oldest request }); console.log(`\nSame requests, same DRAM - FR-FCFS is ${a - b} cycles faster (${Math.round((1 - b / a) * 100)}%).`);

Look at the two "served order" lines: FR-FCFS has quietly sorted the stream into runs of equal rows. Nudge the 0.45 down (more locality) or up (more chaos) and watch the gap move.

Many masters, and the road back

A real controller serves several requestors at once, and they want opposite things. The CPU wants latency: its next hundred instructions are stalled on one load. The GPU and the DMA engines want bandwidth: nobody is blocked on any single beat, but the aggregate rate must be huge. A scheduler that maximises pure row-hit rate will happily serve a thousand-beat GPU streak while the CPU's lone, row-conflicting load ages in the queue. So practical schedulers layer policy on top of FR-FCFS: priority classes per requestor, age counters that escalate starving requests, sometimes per-class bandwidth quotas. Remember this shape — a scheduler is a policy, and every policy chooses losers.

Finally, the road back. Because the controller reorders, read data emerges in schedule order, not request order. Every request carried an ID from the fabric, so each returning beat is tagged with its ID, and the return path either obeys the protocol's per-ID ordering rules or carries a small reorder buffer to restore what each master is owed. Reordering inside, promises kept outside.

per-bank queues expose bank-level parallelism; the address decode splits every request into (bank, row, column);
the timing FSM enforces t_{RCD}, t_{RP}, t_{CAS} (and friends) as per-bank down-counters — no command issues while a counter is live;
FR-FCFS scheduling: ready row-hits first, then oldest — reordering for hit rate and parallelism;
open-page bets on locality (hits), closed-page pre-pays the precharge for scattered traffic;
requestor arbitration balances CPU latency against GPU/DMA bandwidth via priorities and age escalation;
the read-return path tags data with request IDs and restores each master's expected order.

For decades the memory controller didn't live on the processor at all — it sat in the "northbridge", a separate chipset package a few centimetres of motherboard away. Every cache miss took a round trip: off the CPU, across a board-level bus, through the northbridge's controller, out to DRAM, and all the way back. In 2003 AMD's Opteron/Athlon 64 pulled the controller onto the CPU die, and memory latency dropped by tens of nanoseconds overnight — one of the clearest single-change performance wins of that era, and a genuine competitive crisis for Intel, who followed in 2008 with Nehalem. Since then the controller has been a first-class citizen of the SoC, co-designed with the cache hierarchy and fabric — which is exactly why it gets a lesson in a chip-design course: it's your block now, not the motherboard's.

Here is a seductive wrong idea: "delivered bandwidth is the metric, so schedule purely for row hits and long bursts." Build that, and your benchmark bandwidth number is spectacular — while the product stutters. The GPU's endless streaming keeps whole banks' rows open and hit-friendly, so the pure-bandwidth scheduler keeps choosing it; the CPU's single pointer- chasing load — a row conflict, of course — waits, and with it waits every instruction behind it. The display controller, which must refresh the panel on a hard deadline, misses its window and the screen glitches. Nothing is broken in RTL; the policy is broken. A scheduler is a resource allocator, and every allocator has losers — the engineering is choosing them on purpose: priority for latency-critical clients, real-time guarantees for the display, age counters so no request starves, and only then row-hit greed. When a datasheet brags about peak bandwidth, ask the embarrassing question: at what worst-case latency, for whom?