The Block I/O Layer

As an undergraduate you learned that a program calls read(), the CPU issues a command to a device, and eventually an interrupt says "done". That cartoon hides one of the most heavily engineered subsystems in the kernel. Between the \texttt{read()} a program makes and the electrons that finally move on a SATA cable, a single request passes through five or six distinct layers, gets merged with its neighbours, is parked in a queue, handed to a driver, shipped to the device by DMA, and returned to you by an interrupt — or, increasingly, by a busy-polling loop that never sleeps at all.

This lesson maps that path. It is the spine of the whole storage module: every later topic — disk scheduling, the page cache, io_uring — is a modification to one box in the diagram you are about to build. Get the stack straight and the rest of the module becomes "which layer are we optimising, and why?"

The stack, top to bottom

Think of the storage stack as a lift shaft. A request steps in at the top as a byte range in a file and steps out at the bottom as a command to a specific device. Each floor translates the request into the vocabulary of the floor below.

VFS + file system. The Virtual File System gives every file system one common API, then the concrete file system (ext4, XFS, ZFS, Btrfs) turns "byte 9000 of this file" into "logical block 42 of this device".
Page cache. A read is first checked against RAM. On a warm cache the request never descends further — this is why re-reading a file is essentially free. Writes usually land here too, marked dirty, to be flushed later.
Block layer. The heart of this lesson. It packages the work into bio structures, drops them into a request queue, and — crucially — merges and reorders them before they reach the device.
Driver. Speaks the device's command language (SCSI, ATA, NVMe) and sets up the DMA transfer so the device can move data to/from RAM without the CPU copying it.
Device. Performs the physical transfer and signals completion — by an interrupt, or by leaving a flag the kernel polls.

Why the block layer earns its keep: merging and plugging

Suppose a program writes a 64 KiB buffer. The file system chops it into sixteen 4 KiB blocks that happen to be contiguous on disk. If the block layer forwarded sixteen tiny requests, a hard disk would pay a seek and command overhead for each. Instead the block layer coalesces adjacent requests into one large sequential transfer — one seek, one command, sixteen blocks. On a spinning disk this is the difference between a millisecond and a small fraction of one.

To create the opportunity to merge, the block layer uses plugging: when it sees a burst of I/O beginning, it briefly "plugs" the queue — holding requests back for a very short window — so that neighbours arriving microseconds apart can be discovered and fused, then "unplugs" and releases the merged, sorted batch. It is the storage analogue of waiting a beat at a lift so several people ride together instead of sending the car up empty sixteen times.

it turns file-system block requests into bio/request structures on a per-device request queue;
it merges adjacent requests and reorders them (via the I/O scheduler) so the device sees fewer, larger, better-ordered transfers;
it plugs the queue briefly to gather mergeable neighbours, then unplugs;
it hands finished commands to the driver, which programs DMA and takes a completion (interrupt or poll) back up the stack.

Simulate merging

The model below takes a stream of single-block requests and merges any that are contiguous (request for block b can extend a pending request that ends at block b). Watch sixteen little requests collapse into a handful of big ones — and count the device commands saved.

// Block-layer merging: fuse contiguous 4 KiB block requests into large sequential transfers. // Each incoming request is a starting block number for a single 4 KiB block. const incoming = [10, 11, 12, 13, 40, 41, 100, 14, 42, 101, 102]; type Req = { start: number; len: number }; const queue: Req[] = []; for (const blk of incoming) { // Try to extend an existing request that ends exactly at `blk`. const ext = queue.find((r) => r.start + r.len === blk); if (ext) { ext.len++; continue; } // Or one that starts exactly at blk+1 (prepend). const pre = queue.find((r) => r.start === blk + 1); if (pre) { pre.start = blk; pre.len++; continue; } queue.push({ start: blk, len: 1 }); } console.log(`${incoming.length} single-block requests merged into ${queue.length} transfers:`); for (const r of queue) console.log(` blocks ${r.start}..${r.start + r.len - 1} (${r.len} block${r.len > 1 ? "s" : ""}, ${r.len * 4} KiB)`); console.log(`device commands saved: ${incoming.length} - ${queue.length} = ${incoming.length - queue.length}`);

Sync vs async, blocking vs non-blocking

These two axes are constantly confused, so pin them down. Blocking vs non-blocking is about whether the calling thread waits: a blocking \texttt{read()} puts the thread to sleep until data is ready; a non-blocking one returns immediately, perhaps with \texttt{EAGAIN} ("nothing yet"). Synchronous vs asynchronous is about who does the work and when you learn it finished: a synchronous call performs the I/O inline; an asynchronous one submits the request and the completion arrives later, decoupled from the submitting call.

Model	Calling thread	Completion learned via	Example
Synchronous blocking	sleeps until done	the call returns	plain `read()`
Synchronous non-blocking	returns at once	you poll / retry	`O_NONBLOCK` + `epoll`
Asynchronous	returns after submit	a later completion event	io_uring, POSIX AIO

The whole arc of the module bends toward the bottom row: submit many, wait once, be told on completion. That is what lets one core drive a million IOPS to an NVMe drive.

Early machines used programmed I/O: the CPU read the device one word at a time into a register and stored it to memory, byte after byte, burning cycles as a glorified copy machine. Direct Memory Access ended that. The driver hands the device a list of physical memory addresses (a scatter-gather list), says "put the 128 KiB there," and steps away. A dedicated DMA engine moves the bytes straight between device and RAM across the memory bus while the CPU runs other threads. The CPU's only jobs are to set up the transfer and to notice it finished. This is precisely why the interesting cost of an I/O is not the data movement but the per-request overhead — building the command, the doorbell write, the completion — which is exactly what merging and batching attack.

The reflex from undergraduate OS is "interrupts good, polling wasteful" — because for a 10 ms disk seek, sleeping the thread and taking an interrupt is obviously right; you'd waste ten million cycles spinning. But an NVMe SSD can complete a read in ~10 µs. Taking a hardware interrupt costs a context switch, cache pollution, and thousands of cycles — a large fraction of the I/O itself. So Linux's NVMe driver supports hybrid and busy polling (io_poll): for ultra-low-latency devices the CPU spins waiting for completion, because spinning for 10 µs beats the overhead of sleeping and being woken. The right answer flips with the device: interrupt when the wait is long, poll when it is short. Never assume; measure against the device's latency.

Putting the numbers on it

Keep this hierarchy in your head for the rest of the module — every optimisation is a bet about which of these you are paying:

Operation	Rough latency	Relative
DRAM access (page-cache hit)	~100 ns	1×
NVMe SSD read	~10–100 µs	~100–1000×
SATA SSD read	~100 µs	~1000×
HDD random read (seek + rotation)	~10 ms	~100 000×

A page-cache hit is a hundred thousand times faster than a disk seek. That single ratio is why the block layer, the scheduler, and the cache exist at all — and why the next lessons obsess over turning random access into sequential and RAM hits into more RAM hits.