Picture your SoC's expensive out-of-order CPU — the block that ate half the area budget —
spending its evening copying bytes. A network interface delivers a packet; the CPU loops:
load a byte, store a byte, load a byte, store a byte, a hundred million times a second, its
gigahertz pipeline reduced to a bucket brigade. Every cycle spent carrying data is a cycle
not spent computing. This lesson is about the two inventions that free it: giving every
peripheral a clean register-bank programming model so the CPU issues
orders instead of doing chores, and the DMA engine — a dumb,
tireless hardware memcpy that
From the CPU's point of view, every peripheral — UART, timer, SPI controller, the
DMA engine itself — is the same kind of object: a small bank of
| Offset | Register | Read | Write |
|---|---|---|---|
0x0 | CTRL | current config | enable bit, interrupt-enable bit, baud divisor |
0x4 | STAT | TX_FULL, RX_READY flags | — (read-only) |
0x8 | TXDATA | — | push a byte into the transmit FIFO |
0xC | RXDATA | pop the oldest received byte | — (read-only) |
In RTL this is one address decoder and a case statement: a write whose offset is
0x8 pushes the low byte into the TX FIFO; a read at 0x4 muxes the
status flags onto the read-data bus. The deep point is that these are not memory locations
wearing addresses — they are commands and sensors. Writing TXDATA
does something; reading RXDATA consumes something (read it twice and the
second byte is different!). The address map is the peripheral's API, and the datasheet page
listing offsets and bit-fields is its documentation. Every device driver you have ever used
bottoms out in loads and stores to a table exactly like this one.
A byte arrives and sets RX_READY. How does software find out? Option one is
polling: sit in a loop reading STAT until the flag flips. Simple,
lowest-latency, and a perfectly good choice when the wait is short and predictable — but as
a lifestyle it burns the CPU at 100% to watch a flag that flips once a millisecond. Option
two is the
One detail scales badly: a real SoC has dozens of peripherals and the CPU has one or two interrupt inputs. Between them sits the interrupt controller — a block that aggregates all the device lines, latches which ones are pending, applies per-line enables and priorities, and presents the CPU with a single interrupt plus a register that answers "who was it?". The handler reads that register, dispatches to the right driver, and acknowledges the line. (It is, of course, itself just another memory-mapped register bank.)
But interrupts alone don't fix the bucket brigade. A 1 MB/s stream with one interrupt per byte is a million handler invocations a second — the CPU drowns in entry/exit overhead instead of load/store overhead. What we actually want is to hand the whole copy to someone else.
A DMA engine (direct memory access) is a hardware block that does one
thing: copy
Real engines add one more layer of delegation: descriptors. Instead of registers holding a single (SRC, DST, LEN), the driver builds a linked list of little structs in memory — each one a copy command, plus a pointer to the next — and hands the engine the address of the first. The engine fetches each descriptor itself, performs it, and follows the chain. This is scatter-gather: a network packet spread across five non-contiguous buffers becomes one chain of five descriptors and zero CPU copies. The CPU has been promoted from labourer to author of to-do lists.
Step through one transfer. Notice who is doing the work in each phase — the CPU appears only at the two ends:
Here is that diagram as executable behaviour: a register-level model of the engine. The "driver" writes SRC/DST/LEN, sets GO, then polls STAT while the engine moves 4 bytes per cycle behind its back:
Change W to 8 (a wider bus) or LEN to 32 and rerun — the cycle
count follows
The coherence hazard. One honest paragraph, because this bug ships constantly: the DMA engine reads and writes DRAM, but the CPU reads and writes its cache. When DMA deposits a fresh network packet into a buffer, the CPU may still hold stale cache lines for those addresses — and will happily read yesterday's bytes. In the other direction, the CPU may have "written" a buffer that still lives only in a dirty cache line, so the DMA engine transmits garbage from DRAM. Unless your SoC routes DMA through a coherent port on the cache hierarchy, the driver must flush dirty lines before an outbound transfer and invalidate lines before reading an inbound one. This is not an optimisation; it is correctness.
Double-buffering. A transfer you wait for is a transfer that didn't buy
you much. The standard idiom uses two buffers in ping-pong: the DMA engine fills buffer
B while the CPU processes buffer A; at the completion interrupt they
swap. Transfer and computation now fully overlap, and as long as processing a buffer takes
no longer than filling one, the CPU never waits for data. The steady-state period
is
The Commodore Amiga stunned the world with animation its CPU could never have drawn alone — because it wasn't drawing. Alongside the 7 MHz 68000 sat the blitter (from "block image transfer"), a dedicated engine that copied and combined rectangles of screen memory on its own, while separate DMA channels fed audio samples to the speakers and streamed sprites to the display, all stealing bus cycles the CPU wasn't using. Game programmers learned to think like SoC architects: set up the blitter's source, destination and modulo registers, kick it, and go compute the next frame's logic while the hardware painted. Forty years later your phone does exactly this — camera ISP, display controller, audio, radio and storage all run on DMA, and the "computer" is mostly a federation of copy engines with a CPU as their manager. The Amiga was simply early.
The classic DMA bug: build a packet in a buffer, set GO — and transmit garbage. Two versions of the same mistake. First, racing the producer: the code that fills the buffer hasn't actually finished (a function still queued, a store still sitting in a write buffer) when GO is written; the engine, reading at bus speed, overtakes the producer and captures a half-written buffer. Second, racing the cache: the buffer is "written" — but only into dirty cache lines that DRAM has never seen, so the engine faithfully copies the old memory contents. Both failures are intermittent, load-dependent, and invisible in the debugger, because by the time you look, the data has arrived. The discipline is explicit ordering, not hope: finish all writes, execute the cache flush (or use a coherent DMA port), issue a memory barrier so the flush completes before the register write, then set GO. Symmetrically on receive: invalidate before you read, and never touch a buffer the engine still owns.