The Case for Domain-Specific Architectures
For fifty years the deal was simple: wait eighteen months, buy the next chip, and your program ran
faster for free. That deal is off. Dennard scaling
— the rule that let each smaller transistor switch at the same power — ended around 2005,
and with it the free lunch of ever-rising clock speeds. Transistors kept shrinking for another decade,
but every extra one now cost power we could not spend, so much of the chip had to sit dark.
Single-thread performance, which grew about 52\% a year through the 1990s,
slowed to a crawl of a few percent.
So where does speed come from now that the general-purpose treadmill has stalled? Hennessy and Patterson,
in their 2018 Turing-Award lecture, gave the answer a name: a New Golden Age of computer
architecture, built not on faster general chips but on domain-specific architectures (DSAs)
— hardware sculpted to do one class of job extraordinarily well. If you cannot make everything
faster, make the thing that matters faster, and pay for it by refusing to do everything else.
Where a general-purpose CPU spends its energy
Here is the uncomfortable truth a DSA exploits. When a general CPU executes a single
8\text{-bit} addition, the addition itself is nearly free — the arithmetic is
a rounding error. Almost all of the energy is spent getting ready to add: fetching the
instruction, decoding it, renaming registers, checking dependencies, predicting branches, reading the
register file. That overhead is the price of being able to run any program. Rough figures for a
modern out-of-order core:
| Where the energy goes (one integer op) | Approx. energy | Useful? |
|---|
| The arithmetic itself (an 8-bit add) | ~0.03 pJ | yes — this is the work |
| Reading the register file | ~0.1 pJ | overhead |
| Instruction fetch + decode + issue | ~2 pJ | overhead |
| Out-of-order control (rename, wakeup, retire) | ~3–5 pJ | overhead |
The useful arithmetic can be under 1\% of the energy. A DSA throws the other
99\% away by hard-wiring the operation it cares about: no fetch, no decode, no
speculation — just a sea of arithmetic units doing the one thing, fed by a memory system laid out for
exactly that access pattern. The payoff is measured in operations per joule, and DSAs
routinely win by 10\times to 1000\times on their
home turf.
The bargain: generality traded for efficiency
A DSA is not a better CPU — it is a narrower one. It gives up the ability to run arbitrary code
in exchange for doing its chosen workload with far less silicon and energy. The gains come from a handful
of levers, and Hennessy & Patterson name the recipe:
- Use dedicated, local memories managed by software, not general caches — the
compiler knows the access pattern, so hardware need not guess it;
- Spend the saved transistors on more arithmetic units and larger on-chip memory,
not on out-of-order control;
- Exploit the easiest parallelism that matches the domain (wide SIMD / systolic data
flow for matrix work), not hard general-purpose parallelism;
- Shrink the data to the smallest type the domain tolerates
(8\text{-bit} integers or 16\text{-bit} floats
for neural nets), packing more ops into the same joules and bytes;
- Use a domain-specific programming model so the whole software stack targets the
hardware directly.
The zoo of successful DSAs is already all around you: GPUs for graphics and now dense
linear algebra; Google's TPU for neural-network inference and training;
DSPs for signal processing in every phone modem; fixed-function video codecs
that decode 4K H.265 for milliwatts where a CPU would melt; and the image-signal processor behind your
camera. None of them can run a web server — and none of them needs to.
Amdahl never left the room
A DSA speeds up only the fraction of your application that fits the domain. Everything else — the glue
code, the operating system, the data shuffling — runs on the same tired general core. So the whole-system
speedup is governed by the very same
Amdahl's law
that caps every optimisation:
S \;=\; \frac{1}{(1-f) \;+\; \dfrac{f}{A}},
where f is the fraction of runtime the accelerator handles and
A is how much faster it makes that fraction. Drag the accelerator's raw speed
A and watch what happens: even as A screams toward
infinity, the curve slams into the ceiling 1/(1-f). A
100\times matrix engine bolted onto an app that spends only
f = 0.7 of its time in matrix multiply gives barely
3.2\times overall. This is why real accelerator projects fight to grow
f — pulling more of the workload onto the chip — quite as hard as they fight to
grow A.
Worked example: is the accelerator worth it?
Your recommender service spends f = 0.8 of its time doing the matrix maths a
new DSA can run A = 50\times faster. The overall speedup is
S = \frac{1}{(1-0.8) + 0.8/50} = \frac{1}{0.2 + 0.016} = \frac{1}{0.216} \approx 4.6\times.
A 50\times engine bought you 4.6\times — because the
stubborn 20\% that stayed on the CPU now dominates. The lesson every
accelerator architect learns: profile first, and only build silicon for a fraction big enough to be
worth it. Run the numbers for a range of designs below.
// Whole-system speedup from an accelerator (Amdahl): f = fraction accelerated, A = its raw speedup.
function systemSpeedup(f: number, a: number): number {
return 1 / ((1 - f) + f / a);
}
// A fixed, very fast accelerator (A = 50x) applied to workloads that spend
// more and more of their time in the accelerated kernel.
const A = 50;
console.log(`Accelerator raw speedup A = ${A}x. Ceiling for each f is 1/(1-f):`);
for (const f of [0.5, 0.7, 0.8, 0.9, 0.95, 0.99]) {
const s = systemSpeedup(f, A);
const ceiling = 1 / (1 - f);
console.log(`f = ${f}\tsystem = ${s.toFixed(1)}x\t(ceiling ${ceiling.toFixed(1)}x)`);
}
The GPU was designed for one domain — shading millions of pixels — which happens to be embarrassingly
parallel arithmetic over big arrays. In 2012 a network called AlexNet was trained on two gaming
GPUs and crushed the image-recognition record, and researchers realised that the same wide-SIMD hardware
that shaded pixels was almost perfect for the matrix multiplies inside a neural net. The GPU was an
accidental DSA for deep learning, and NVIDIA spent the next decade making it a deliberate one
(adding tensor cores — little systolic matrix engines) until AI hardware became its main
business. The moral: pick your domain well, because a domain can grow far beyond what you imagined.
The seductive mistake is to read "100\times faster" and imagine your whole
program running 100\times faster. It won't. A DSA accelerates only the slice of
work that matches its domain; anything off-domain runs on the host CPU, often slower than a
good general core would, and the data must be shipped back and forth across a bus. Two costs bite: Amdahl
caps the win at 1/(1-f), and programmability — you must
rewrite your software in the DSA's restricted model, and if the workload later changes, the fixed silicon
cannot follow. Generality is the thing you sold; don't be surprised when you miss it.
The road ahead
The rest of this module is the New Golden Age in miniature. We build the archetypal DSA — the
systolic array at the heart of the TPU — then study the networks that
stitch thousands of these chips together, and finally zoom all the way out to the
warehouse-scale computer, where the whole building is the machine. Every one of them is
the same bet: give up generality you don't need, and buy efficiency you do.