fsync, Durability and Write Barriers

Every previous lesson kept the file system consistent. This one is about a different, sharper promise: durability — the guarantee that data you wrote is actually on stable media and will survive a power failure. The two are constantly confused, and the confusion has cost real companies real data. Here is the uncomfortable truth: when your program's \texttt{write()} call returns successfully, your data is almost certainly not on the disk. It is sitting in volatile RAM, and a power cut in the next instant would erase it without a trace — even though \texttt{write()} told you it succeeded.

That gap between "the write call returned" and "the bytes are on the platter" is the subject of this lesson. To close it deliberately you use \texttt{fsync}, and to make \texttt{fsync} actually work the operating system must punch a write barrier through several layers of caching all the way down to the spinning rust or flash. Getting this right is subtle enough that it has produced famous data-loss bugs in ext3, ext4, and countless applications.

The page cache: why write() is a lie of omission

For speed, \texttt{write()} does not go to the disk. It copies your bytes into the page cache — an area of kernel RAM — marks that page dirty, and returns immediately. Your program races on, believing the write is done, while the actual disk I/O happens later, asynchronously, when the kernel's writeback threads get around to it (in Linux, dirty pages can linger up to ~30 seconds by default). This buffering is a huge performance win — it batches writes, lets the elevator reorder them, and absorbs rewrites of the same block — but it means the window between "wrote" and "safe" can be tens of seconds wide.

write() — copies data into the page cache and returns; not durable;
fsync(fd) — force all dirty pages of this file, and its metadata, to stable storage, issuing a cache flush; returns only when durable;
fdatasync(fd) — like \texttt{fsync} but skips inessential metadata (e.g. timestamps), so it can avoid an extra metadata write — faster when only the data must be safe;
write barrier / FLUSH / FUA — the low-level command that forces the drive's own cache out to the physical medium (or writes straight through it).

Two volatile caches, one stable medium

There is not one cache in the way but two. Data leaving the page cache lands in the disk's own write cache — a chunk of RAM on the drive itself — which reports the write "complete" the moment it arrives there, long before the head has actually put it on the platter. So even after the kernel has flushed the page cache, the bytes can still evaporate on power loss. Only a FLUSH CACHE command (or writing with the FUA — Force Unit Access — bit set) drives them through that last volatile layer onto stable media. Trace the whole path:

\texttt{fsync}'s real job is to push a barrier through every stage here — flush the relevant dirty pages out of the page cache and issue the drive FLUSH — so that when it returns, the data has genuinely crossed the volatile/stable line.

Where does a crash bite?

The model below tracks a write as it moves down the stack and asks, at each stage, "if the power failed now, would this survive?" The answer is no until the very last stage — which is exactly why an explicit \texttt{fsync} is the only way to know your data is safe.

// Where does data live as it flows down the write path, and does it survive a crash at each stage? type Stage = "page-cache (kernel RAM)" | "disk-cache (drive RAM)" | "platter (STABLE)"; const survives = (s: Stage): boolean => s === "platter (STABLE)"; function report(label: string, s: Stage): void { console.log(`${label.padEnd(26)} data in ${s.padEnd(24)} survives crash? ${survives(s) ? "YES" : "no"}`); } // App just calls write() and moves on — no fsync. report("after write()", "page-cache (kernel RAM)"); // Background writeback fires seconds later; still only in the drive's volatile cache. report("after writeback", "disk-cache (drive RAM)"); // The app instead calls fsync(): forces writeback AND a FLUSH/FUA barrier to the platter. report("after fsync() + FLUSH", "platter (STABLE)"); console.log("\nLesson: only fsync()+FLUSH crosses the volatile→stable line.");

The atomic-replace pattern — and its classic trap

The safe way to update a config file so a reader never sees a half-written mess is write-temp-then-rename, because \texttt{rename} over an existing name is atomic:

write the new contents to \texttt{file.tmp};
\texttt{fsync(file.tmp)} — make the data durable;
\texttt{rename(file.tmp, file)} — atomically swap it into place;
\texttt{fsync(directory)} — make the rename itself durable.

Miss step 2 and a crash can leave the renamed file full of garbage. Miss step 4 — the one everyone forgets — and a crash can lose the rename, leaving you with the old file or no file, even though the data was safe. Durability of the data and durability of the directory entry are separate fsyncs, because the file and its parent directory are separate on-disk objects.

For years, applications got away with skipping \texttt{fsync} after the rename trick, and nobody noticed. Why? ext3's default ordered mode had an accidental property: because it flushed all pending data on every metadata commit (every few seconds), the data usually hit the disk right around the rename anyway. Programs were relying on a durability guarantee the standard never made. Then ext4 arrived with delayed allocation — it deliberately held data in the page cache longer to pick better block placement — and suddenly the window widened to tens of seconds. Users who crashed after replacing a file found it truncated to zero bytes: the rename had committed but the data had not. The uproar was so loud that ext4 added heuristics to auto-flush data on a rename-over-existing-file, papering over the buggy apps. The real lesson, seared into a generation of systems programmers: if you did not call \texttt{fsync}, you have no durability guarantee — full stop — no matter what the file system happened to do for you yesterday.

Two promises get tangled here. Ordering means "write B never becomes durable before write A" — it constrains the sequence in which things reach stable storage (journaling leans on this: the commit record must not precede the journal blocks). Durability means "this specific write is on stable media now." A write barrier enforces ordering; \texttt{fsync} enforces durability (and, as a side effect, ordering up to that point). Confusing them leads to two opposite mistakes: thinking your data is safe because writes were ordered (they may all still be in volatile cache), or calling \texttt{fsync} after every tiny write and destroying throughput because each one forces a full drive FLUSH — often milliseconds, an eternity. The art is to fsync at the transaction boundaries that actually matter (a committed database row, a saved document), and let everything in between stream through the caches. Databases fsync once per group-commit for exactly this reason.

The module, closed

You now have the full stack. The VFS gives one interface to many file systems; inodes and allocation lay bytes on disk; crash consistency, journaling, log-structured and copy-on-write designs keep the structures coherent across failures; and \texttt{fsync} with write barriers finally lets an application demand that a specific write is durably saved. Consistency keeps the file system alive; durability keeps your data alive — and only you, by calling \texttt{fsync} at the right moment, can ask for the second.