The Limits of ILP

We have spent this whole module building a magnificent machine for extracting instruction-level parallelism (ILP): dynamic scheduling, register renaming, wide superscalar issue, reorder buffers, world-class branch prediction, and deep speculation. Each trick squeezed out a bit more overlap. So here is the uncomfortable question that closes the module: how much parallelism is actually left to find? If we keep making the machine wider and deeper, does IPC keep climbing?

The answer, established by a famous line of "limit studies" in the early 1990s, is a hard no. Even an idealised machine — perfect branch prediction, infinite renaming, unlimited execution units — extracts only a modest amount of parallelism from typical integer programs. Real code has a ceiling on how much can happen at once, and we are already close to it. This is the ILP wall, and it is one of the two walls that ended the single-core era.

Diminishing returns, made visible

Widening the issue width raises the ceiling on IPC, but real IPC flattens out fast — the code simply doesn't contain enough independent instructions to fill ever-more ports. The curve below is the signature shape of the whole module: steep gains from 1 to 4, then a long, disappointing plateau. Doubling from 8-wide to 16-wide barely moves sustained IPC, while (from the superscalar lesson) the dependency-check hardware cost keeps rising as w^2.

Where the marginal IPC gain falls below the marginal cost — power, area, clock-period pressure — you should stop widening. For general-purpose integer code that crossover sits around 4-wide, which is why mainstream cores clustered there for two decades. Apple's very wide cores (8-wide decode) are the exception that proves the rule: they only pay off alongside enormous ROBs, superb prediction, and carefully tuned workloads.

Three walls hold IPC down

Why can't we find more parallelism? Three fundamental limits, none of which more transistors can repeal:

Limit	What it is	Why hardware can't beat it
True (RAW) dependences	real data flows from one instruction to the next in a chain	renaming removes false deps, but a genuine a\to b\to c chain is inherently serial — you cannot compute a value before its inputs exist
Control (branch) limits	to run far ahead you must predict many branches correctly in a row	even 99% per-branch accuracy compounds: 20 branches deep, 0.99^{20}\approx 82\% of paths survive — the speculation window is bounded
Memory limits	loads may miss in cache and take hundreds of cycles; addresses may alias	the ROB can only hold so many in-flight instructions; a long-latency miss eventually fills it and everything stalls

Notice these are properties of the program, not the chip. A pointer-chasing linked-list traversal is almost perfectly serial — each load's address depends on the previous load's result — and no amount of issue width helps. The available ILP is baked into the algorithm.

even with idealised hardware, typical integer code exposes only modest ILP (often a single-digit IPC);
the limits are true data dependences, branch predictability over long windows, and memory latency + finite in-flight window;
widening issue gives diminishing IPC returns while dependency-check cost grows as w^2, so ~4-wide is the general-purpose sweet spot;
these limits belong to the program, so more transistors on one core can't remove them.

Watch the branch window collapse

The control limit is worth feeling numerically. To keep a big out-of-order window full you must be speculating past many branches at once, and the probability that all of them are predicted correctly falls geometrically:

// How far ahead can we usefully speculate? The chance ALL of the next k branches // are predicted right is accuracy^k -- it decays fast even at high accuracy. function survives(accuracy: number, k: number): number { return Math.pow(accuracy, k); } for (const acc of [0.95, 0.99, 0.997]) { const line = [4, 8, 16, 32] .map((k) => `${k}:${(100 * survives(acc, k)).toFixed(0)}%`) .join(" "); console.log(`per-branch ${(acc * 100).toFixed(1)}% -> window depth ${line}`); } console.log("Even at 99%, 32 branches deep only ~72% of speculative windows are on the right path.");

This is why prediction accuracy has to be so absurdly high, and why it still doesn't buy unlimited parallelism: the further you speculate, the more likely something ahead is wrong, capping the useful window regardless of how wide you build the back end.

The second wall: power

The ILP wall alone might have been survivable — but it arrived hand-in-hand with the power wall. For decades, Dennard scaling let each transistor generation run faster at constant power density. Around 2004 that broke: clock frequencies stopped climbing because faster switching meant untenable heat. So the two classic single-core levers both jammed at once — you couldn't clock much higher (power wall), and you couldn't get much more IPC per core (ILP wall). The iron law's first two terms had hit diminishing returns simultaneously.

A wider, more speculative core also spends a lot of energy on work it throws away — mispredicted paths, squashed speculation — so pushing ILP harder gets worse on the very metric (performance per watt) that now mattered most. The economics inverted: the marginal transistor was better spent elsewhere.

Because the industry changed the question. Once one core hit the ILP and power walls, architects stopped asking "how do I make this core wider?" and started asking "how do I put more cores on the die?" If you can't extract more parallelism from one instruction stream, run several streams at once — thread-level parallelism (TLP). This is the great pivot to multicore around 2005: dual-core, then quad, then the dozens of cores in a modern server. It shoves the parallelism problem up to the software — now you must write threads — but it sidesteps both walls at once. Your laptop is faster because it has more cores and better accelerators, not because each core is dramatically wider than a 2010 one.

The ILP wall is a statement about general-purpose, branchy, pointer-heavy integer code — compilers, browsers, databases. Plenty of other workloads are embarrassingly parallel: dense linear algebra, graphics, neural-network math, signal processing. Those aren't limited by ILP at all — they're limited only by how many arithmetic units you can feed, which is exactly why GPUs, SIMD/vector units, and AI accelerators exist and scale to thousands of lanes. Don't over-generalise the ILP wall into "parallelism is dead." The right lesson is narrower and sharper: a single sequential instruction stream has limited parallelism, so we extract parallelism at other granularities instead (threads, data, accelerators).

Where the module ends, and the next begins

This module took one instruction stream and wrung out every drop of overlap the hardware could find — an extraordinary engineering achievement that hides behind the simple fact that your code "just runs fast." But we've now met the ceiling: true dependences, branch windows, and memory latency cap how much a single core can do, and the power wall forbids simply clocking higher. Those two walls, arriving together around 2004–2005, are why every processor you own is multicore. The story of computer architecture now moves up a level — from parallelism within one instruction stream to parallelism across many — which is exactly where the next module picks up.