Persistent Memory and NVMe

For fifty years the storage hierarchy had a comfortable shape: fast, small, volatile RAM on one side; slow, huge, durable disk on the other; and a yawning gap of five or six orders of magnitude between them. Every operating system was designed around that gap — the block layer, the page cache, the whole ceremony of copying data in fixed-size blocks between two utterly different worlds. Two technologies are quietly collapsing that gap, and with it some assumptions baked deep into the kernel: NVMe and persistent memory.

The theme of this closing storage lesson is that when the device stops being slow, the software becomes the bottleneck — and the OS abstractions built to hide a slow disk start to get in the way.

The gap, and the newcomer that fills it

Look at the ladder of latencies. The old world had nothing between DRAM (tens of nanoseconds, volatile) and an SSD (tens of microseconds) — a jump of roughly 1000\times. Persistent memory (most famously Intel's now-discontinued Optane, and its successors) sits right in that gap: byte-addressable like RAM, only a few times slower than DRAM, sitting on the memory bus — and it keeps its contents when the power dies.

That single new rung breaks a distinction the whole OS was built on: memory is fast-but-forgetful, storage is durable-but-distant. Persistent memory is both fast and durable, and the kernel has no natural home for it.

NVMe: when the disk is fast, the old interface is the bottleneck

The SATA/AHCI interface that served spinning disks assumed a single, slow, serial device: one command queue, 32 entries deep, one lock. That was fine when a seek cost ten milliseconds — the software overhead was invisible against the mechanical wait. But a modern flash SSD can service hundreds of thousands of I/Os per second, and now the driver's lock and single queue are the wall.

NVMe (Non-Volatile Memory Express) is the storage interface redesigned for parallel, low-latency devices. Its key idea is many deep queues — up to 64\text{,}000 queues of 64\text{,}000 commands each — and, crucially, a queue per CPU core. Each core submits to its own queue with no shared lock, so the storage stack scales with cores the way the device does. The lesson mirrors scalable synchronization: kill the shared lock, give each core its own path.

	AHCI / SATA	NVMe
Command queues	1	up to 64K
Queue depth	32	up to 64K
Per-core queues	no (shared lock)	yes (lock-free path)
Register writes per command	several (MMIO, slow)	one doorbell

Byte-addressable persistence — and the ordering trap

Persistent memory lets a program treat durable data as ordinary memory: map it in, and read and write it with plain \texttt{load} and \texttt{store} instructions — no \texttt{read()}, no block layer, no page cache. Linux exposes this as DAX (Direct Access): a file on a PM device is mapped straight into the address space, and the page cache is bypassed entirely, because caching durable memory in other memory would be pointless.

But there is a subtle, dangerous catch. When your \texttt{store} executes, the data lands in the CPU's cache — which is still volatile. It is not truly persistent until it has been flushed out to the PM device. So durable programming on PM needs explicit cache-line flushes (\texttt{CLWB}) and store fences (\texttt{SFENCE}) in the right order, or a crash can leave your "persistent" data structure half-written. The durability vs ordering problem you met for file systems reappears here at the granularity of a single cache line.

// The persistence ordering trap on PM: a store reaches durability only after a flush + fence. // Model a crash at each point and check whether the durable value + its "valid" flag are consistent. type PM = { value: number; valid: boolean }; function commitCorrectly(pm: PM): void { pm.value = 42; // store to PM (still in volatile CPU cache) // CLWB(value); SFENCE; // <-- flush the value BEFORE marking it valid pm.valid = true; // publish // CLWB(valid); SFENCE; } // If a crash lands between "value" and "valid" WITHOUT the flush, a reader may see valid=true // but the value never reached the device. Ordering the flushes is what prevents that torn state. const cases = ["crash after value, before flush", "crash after flush+fence, before valid", "clean commit"]; for (const c of cases) { const durable = c === "clean commit"; console.log(`${c}: reader sees consistent state = ${durable}`); } console.log("Lesson: on PM, YOU order persistence with CLWB/SFENCE — the cache is volatile.");

Why this reshapes the OS

Put the two together and the kernel's storage stack looks increasingly like overhead. If the device answers in a microsecond, the hundreds of cycles spent in the system call, the block layer, and the interrupt now dominate the total. That is why the modern answer is often to get the kernel out of the data path: poll instead of interrupt, batch with io_uring, or bypass the kernel entirely with user-space drivers (SPDK). Fast hardware turns a decades-old software stack into the slow part.

NVMe replaces the single-queue AHCI interface with thousands of deep, per-core queues, so the storage stack scales with cores and the device's own parallelism;
Persistent memory is byte-addressable, near-DRAM-speed, and durable — it fills the DRAM↔SSD latency gap and breaks the "memory is volatile, storage is durable" divide;
DAX maps PM straight into a process and bypasses the page cache;
persistence on PM requires explicit cache-line flush + fence ordering — the CPU cache is volatile, so a store is not durable until flushed;
when the device is this fast, software overhead dominates — hence kernel-bypass, polling, and batching.

Optane was a genuine marvel — but it was caught in a vice. It had to be cheaper than DRAM to be worth buying for capacity, yet expensive enough to fund a whole new fabrication process; and it had to be enough faster than a good NVMe SSD to justify a new programming model that almost no software adopted. Rewriting applications to use durable loads and stores with manual flushing turned out to be hard, and NVMe SSDs kept getting faster and cheaper underneath it. Intel wound Optane down in 2022. The ideas, though — byte-addressable persistence, DAX, CXL-attached memory pools — are very much alive, and the "memory is durable now" problem it posed to OS designers hasn't gone away.

The single most common bug in PM programming is assuming that because you wrote to a persistent-memory address, the data is safe. It is not — your \texttt{store} sits in the volatile CPU cache and may not reach the device for a long time, or ever, if the line is simply overwritten. Only an explicit flush (\texttt{CLWB}/\texttt{CLFLUSHOPT}) followed by a fence guarantees it. And ordering matters: if you publish a "valid" flag before the data it refers to has been flushed, a crash yields a structure that looks valid but points at garbage. Durable data structures need the same write-ahead discipline as a journal, just at cache-line granularity.