Protection and Isolation Models

A modern machine runs your bank's app, a stranger's JavaScript, a background updater, and the kernel itself — all on the same silicon, at the same time. The kernel is the referee, but a referee needs a rulebook: a precise statement of who is allowed to do what to which resource. This lesson is about that rulebook — the mathematical object underneath every permission system you have ever met, and the two rival ways real systems store it.

The vocabulary is deliberately abstract so it covers everything. A subject is an active entity that does things — a process, a user, a thread. An object is a passive thing that is acted upon — a file, a socket, a page of memory, another process. A right (or permission) is a verb the subject may apply to the object — read, write, execute, send-signal. Protection is nothing more than deciding, for every (subject, object) pair, which verbs are allowed.

The protection matrix

Lay every subject down the side and every object across the top, and the complete security state of a system is one giant table — the access-control matrix of Lampson (1971). The cell in row s, column o holds the set of rights that subject s holds over object o:

A[s, o] \;\subseteq\; R, \qquad R = \{\,\texttt{read},\ \texttt{write},\ \texttt{exec},\ \texttt{own},\ \dots\}.

"May process s write file o?" is answered by one lookup: is \texttt{write} \in A[s,o]? Every access decision the OS makes is, conceptually, exactly this test. The matrix is the ground truth; everything else in this lesson is an encoding of it that fits in real memory.

And it must be encoded cleverly, because the full matrix is hopeless to store literally. A server with 10{,}000 users and 1{,}000{,}000 files has a matrix of 10^{10} cells, almost all of them empty. So no real system stores the whole grid. Instead they store it sliced — and the direction of the slice is the single most important design decision in access control.

Two ways to slice the same matrix

There are only two natural ways to cut a table into storable pieces: by column, or by row.

Watch the same matrix get sliced both ways. The highlighted column is one file's ACL; the highlighted row is one process's capability list.

ACLs vs capabilities — the trade-offs

The choice of slice direction is not cosmetic; it decides which operations are cheap and which are agonising. Auditing "who can touch this file?" is trivial with an ACL (read the column) and brutal with capabilities (search every subject's tickets). Auditing "what can this process touch?" is the reverse. And revocation — taking a right back — is the hardest problem in capability systems, because the tickets have already scattered into subjects' hands.

QuestionACL (column-wise)Capability (row-wise)
Who can access object o?easy — read o's listhard — scan all subjects
What can subject s access?hard — scan all objectseasy — read s's tickets
Grant a rightedit the object's listhand over a copy of the ticket
Revoke a rightedit the object's list (immediate)hard — tickets already delegated
Check on accessverify caller's identity vs listjust present the ticket (no identity lookup)
Delegation to a helperawkward (confused-deputy risk)natural — pass the ticket along
Real examplesUnix/POSIX perms, NTFS, SELinux labelsfds, seL4, Capsicum, KeyKOS, WebAssembly imports

Capabilities elegantly dodge the confused-deputy problem — a privileged program tricked into misusing its authority on a caller's behalf — because the caller must hand over the exact ticket, so the deputy can never exceed what it was given. This is why capability designs (seL4, Fuchsia's Zircon, Capsicum on FreeBSD) keep reappearing at the security-critical edge of the field.

The permission check, in code

Both encodings answer the very same question — \texttt{write} \in A[s,o]? — but arrive at the answer from opposite directions. Here they are side by side: the ACL walks the object's list looking for the subject; the capability check just asks whether the subject already holds a matching ticket. Run it.

// The SAME access-control matrix, encoded two ways. Both answer: may `subject` do `right` to `object`? type Right = "read" | "write" | "exec"; // (1) ACL encoding: each OBJECT stores who may do what. const acl: Record<string, Record<string, Right[]>> = { "report.txt": { alice: ["read", "write"], bob: ["read"] }, "keys.pem": { alice: ["read"] }, }; function aclCheck(subject: string, right: Right, object: string): boolean { const entry = acl[object]?.[subject] ?? []; // find the subject in the object's column return entry.includes(right); } // (2) Capability encoding: each SUBJECT holds unforgeable tickets (object + rights). interface Cap { object: string; rights: Right[]; } const bags: Record<string, Cap[]> = { alice: [ { object: "report.txt", rights: ["read", "write"] }, { object: "keys.pem", rights: ["read"] } ], bob: [ { object: "report.txt", rights: ["read"] } ], }; function capCheck(subject: string, right: Right, object: string): boolean { const ticket = bags[subject]?.find((c) => c.object === object); // find the object in the subject's row return ticket?.rights.includes(right) ?? false; } for (const [s, r, o] of [["alice","write","report.txt"],["bob","write","report.txt"], ["bob","read","keys.pem"],["alice","read","keys.pem"]] as const) { const a = aclCheck(s, r, o), c = capCheck(s, r, o); console.log(`${s} ${r} ${o}? ACL=${a} CAP=${c} ${a === c ? "(agree)" : "(!!)"}`); }

Least privilege — the master principle

Whatever encoding you pick, how much should each cell contain? Saltzer and Schroeder's 1975 answer still governs the field: the principle of least privilege. Every subject should hold the minimum set of rights needed to do its job, for the shortest time, and nothing more. A PDF viewer needs to read one file and draw pixels; it has no business opening network sockets or reading your SSH keys. If it is ever compromised, the blast radius is bounded by exactly the rights it held at that instant.

Least privilege is why the whole rest of OS security exists: dropping capabilities after start-up, seccomp syscall filters, privilege separation, sandboxes. It is also violated by the most common Unix idiom of all — a program running as \texttt{root} holds every right, so a single bug is total compromise. The historical fix, breaking a big privileged daemon into a tiny privileged core plus a large unprivileged worker (as OpenSSH did in 2002), is least privilege applied with a chainsaw.

Who sets the rights? DAC vs MAC

There is a second, orthogonal axis: who is allowed to change the matrix?

Linux ships MAC as a Linux Security Module (LSM) layered after the ordinary DAC check: a request must pass the classic permission bits and the MAC policy. The two big implementations are SELinux and AppArmor. SELinux uses type enforcement: every process runs in a domain (e.g. \texttt{httpd\_t}) and every object carries a type (e.g. \texttt{httpd\_sys\_content\_t}), and a rule base spells out exactly which domains may touch which types. If no rule permits it, it is denied — default-deny. AppArmor takes a friendlier path: it confines programs by pathname profiles rather than labels. This is precisely why a web server broken into via a bug still cannot read \texttt{/etc/shadow}: DAC might allow it after a mistake, but the MAC policy never will.

Rings and the reference monitor

All of this rests on a hardware foundation you already know: the CPU's protection rings. x86 defines four (ring 0 = kernel, ring 3 = user; rings 1–2 largely unused; a hypervisor sits in a "ring −1"). The privilege bit is what makes the whole matrix enforceable rather than merely advisory — a ring-3 process physically cannot rewrite the page tables or the ACLs that constrain it.

The component that actually performs the check — the code that consults the matrix on every access — is the reference monitor. Anderson's 1972 report gave it three non-negotiable properties, and they are the bar every OS security mechanism is measured against:

The trusted computing base (TCB) is the set of all the hardware and software you must trust for security to hold — the rings, the kernel, the reference monitor. Every idea in this module is, at heart, an attempt to make that TCB smaller and its guarantees stronger.

Pure capability machines are almost as old as Unix — Cambridge's CAP computer, the Plessey System 250, later KeyKOS and EROS — and researchers have adored them for fifty years: no ambient authority, no confused deputies, clean delegation. Yet ACLs conquered the world. Why? Partly inertia and the sheer gravitational pull of Unix, but partly a genuine mismatch: humans reason naturally about "who can see my folder?" (an ACL question) and much less naturally about tracking a swarm of unforgeable tickets and revoking them later. Revocation, in particular, is a capability system's Achilles' heel — once you have handed out a ticket you cannot easily claw it back. The modern verdict is that both were right: your laptop's filesystem is ACL-based, but the sandbox around your browser tab, a WebAssembly module's imports, and the seL4 microkernel are all pure capabilities. The industry quietly adopted capabilities exactly where least privilege matters most.

A classic misreading is to treat \texttt{rwx} bits as an unbreakable security boundary. They are discretionary: the file's owner can change them at will, any program running as you inherits all your authority (ambient authority), and the superuser bypasses the checks entirely — \texttt{root} reads a \texttt{0600} file with a shrug. So "I set it to \texttt{600}, it's safe" is only true against other unprivileged users, and only until some process of yours is compromised. If you need a boundary that even root and even the file owner cannot cross, you need mandatory access control (SELinux/AppArmor) or a capability system — a policy nobody can relax at their discretion. DAC bits and MAC policy live in different columns of the same battlefield; don't mistake one for the other.