Union and Overlay Filesystems

A container image for a Node.js app might be 900 MB. You run 50 copies of it. Naively that is 45 GB of disk and a fresh multi-second copy every time you start one. In reality Docker starts all fifty in well under a second and they share almost all of that 900 MB on disk. The magic is a union filesystem, and on Linux today that means overlayfs — a VFS-level filesystem that merges several directory trees into one without copying them.

This is the lesson that explains what a container "image" actually is and why building and shipping them is so fast. The answer is a beautiful application of an old idea — copy-on-write — to filesystem layers: a stack of read-only layers shared by everyone, plus one thin writable layer per container, all fused into a single directory the process sees as its normal root.

Layers, stacked and merged

An image is not a monolithic disk blob; it is an ordered stack of layers, each layer a set of file changes (a tarball of "these files were added/changed/deleted"). Each line in a \texttt{Dockerfile} — \texttt{FROM ubuntu}, \texttt{RUN apt install}, \texttt{COPY . /app} — produces one layer. overlayfs's job is to take that stack and present a single merged view. Its vocabulary is four directories:

lowerdir — one or more read-only layers (the image), searched top-to-bottom;
upperdir — the single writable layer for this container's changes;
merged — the unified view the process actually sees as /;
workdir — an empty scratch dir overlayfs needs internally for atomic operations.

A lookup for a file walks the stack from the top: if the upperdir has it, you get that; otherwise overlayfs falls through to the first lowerdir that has it. Same-named files in higher layers shadow lower ones. That is the whole "union" semantics.

The overlay stack

Picture the image as a pile of transparent sheets. Look down through them and you see the topmost mark at each position; the writable sheet on top is where this container scribbles. Everyone else's pile rests on the exact same lower sheets — read-only, so sharing them is safe.

Because the lower layers are immutable, the kernel can point a thousand containers at the same on-disk bytes for \texttt{FROM ubuntu}. The image is downloaded, decompressed, and cached once; each container adds only its own tiny upperdir. That is the deduplication that makes the "45 GB" collapse to "900 MB shared plus a few MB each".

Copy-up: the moment a shared file becomes private

Reads are free — they fall through to the shared lower layers. The interesting event is the first write to a file that only exists in a read-only layer. overlayfs cannot modify the lower layer (it is shared and immutable), so it performs a copy-up: it copies the whole file up into the writable upperdir, and the write — and every future access — goes to that private copy. The lower original is untouched and still shared by everyone else.

This is copy-on-write applied at file granularity: pay the copy cost only when, and only for the files, you actually mutate. A container that only reads its image writes nothing to disk at all. One that edits a single config file copies up exactly that one file. Deleting a file that lives in a lower layer is handled with a whiteout — a special marker in the upperdir that hides the lower entry without touching it.

// A toy overlayfs: read falls through RO layers; the FIRST write copies the file UP. // Deletes leave a whiteout marker. Nothing here is a real FS — it just narrates the semantics. const lower: Record<string, string> = { // shared read-only image layers "/etc/hosts": "127.0.0.1 localhost", "/app/config": "mode=prod", "/usr/bin/node": "<binary>", }; const upper: Record<string, string> = {}; // this container's private writable layer const whiteout = new Set<string>(); // files "deleted" from the merged view function read(path: string): string { if (whiteout.has(path)) return `ENOENT (whited-out)`; if (path in upper) return `${upper[path]} [from upperdir]`; if (path in lower) return `${lower[path]} [from lowerdir — shared, no copy]`; return "ENOENT"; } function write(path: string, data: string): void { if (!(path in upper) && path in lower) { console.log(` copy-up: ${path} copied lower → upper before first write`); } upper[path] = data; whiteout.delete(path); } function remove(path: string): void { delete upper[path]; if (path in lower) whiteout.add(path); // hide the lower version } console.log("read /etc/hosts :", read("/etc/hosts")); // served from lower, shared write("/etc/hosts", "127.0.0.1 localhost\n10.0.0.5 db"); // triggers copy-up console.log("read /etc/hosts :", read("/etc/hosts")); // now from upper console.log("read /app/config:", read("/app/config")); // still shared, never copied remove("/usr/bin/node"); console.log("read /usr/bin/node:", read("/usr/bin/node")); // whited out console.log(`\nupperdir now holds ${Object.keys(upper).length} file(s); the rest stays shared on disk.`);

Content addressing — why layers dedup and cache

How does Docker know two images share the \texttt{FROM ubuntu:22.04} layer, so it stores and pulls it once? Each layer is content-addressed: its identity is the cryptographic hash (a SHA-256 \texttt{digest}) of its contents. Two layers with identical bytes have identical digests and are therefore the same object — stored once, pulled once, cached once, whether they appear in one image or a hundred. This is the same content-addressed idea behind Git objects.

Content addressing also powers the build cache. When you rebuild an image, Docker hashes each build step's inputs; if nothing changed, the step's layer digest is unchanged and the cached layer is reused instead of re-run. This is why ordering a \texttt{Dockerfile} from least- to most-frequently-changed (dependencies before your source) makes rebuilds fast: the expensive early layers stay cache-hits.

An image is an ordered stack of read-only layers; a running container adds one thin writable layer (upperdir) on top.
overlayfs merges lowerdir(s) + upperdir into one merged view; a lookup falls through top-to-bottom, higher layers shadow lower ones.
Copy-up: the first write to a lower-layer file copies it into the upperdir (copy-on-write at file granularity); deletes use a whiteout marker.
Layers are content-addressed (hash of contents), so identical layers are stored, pulled, and cached once and shared across all containers — the source of both dedup and the build cache.

Because starting it copies almost nothing. The gigabyte of read-only layers is already on disk from the \texttt{docker pull} and is shared — the runtime just mounts an overlay that points the new container's lowerdir at those existing layers and gives it a brand-new, empty upperdir. Creating an empty directory and mounting an overlay is a handful of syscalls, not a gigabyte copy. The container then \texttt{exec}s its process against that merged view. All the "weight" of the image was paid once, at pull time; startup is nearly free. This is the storage half of why containers cold-start in milliseconds — the compute half was the shared kernel.

A classic production mistake: an app writes its database or user uploads into the container's filesystem, the container is later replaced (a deploy, a crash, a reschedule), and the data vanishes. The upperdir lives and dies with the container — remove the container and its writable layer is discarded. overlayfs's copy-on-write is designed for image state, not application state. Anything that must survive the container's lifetime belongs in a volume or bind mount — real storage mounted through the overlay, outside the layer stack. Rule of thumb: treat the container filesystem as scratch, and put durable data somewhere the copy-up machinery never touches. Also beware write-heavy workloads: every first-write triggers a full-file copy-up, so churning large files on the upperdir is slower than on a native filesystem — another reason to use a volume.