Every defence in this lesson exists because of one uncomfortable fact: the system-call interface is a doorway that faces a hostile world. On the other side of it is the most privileged code on the machine, and pushing on the door is every process you run — including the ones a stranger's website just started inside your browser. The kernel exposes roughly 350 to 450 syscalls on Linux, plus \texttt{ioctl}s numbering in the thousands, plus every parser that touches a packet, a filesystem image, or a USB descriptor. That entire reachable frontier is the attack surface, and an attacker needs just one exploitable bug anywhere on it.

This is the story of an arms race. For each class of bug, defenders invented a mitigation; for each mitigation, attackers found a bypass, which provoked the next mitigation. Nobody has "won" — modern exploits chain several bypasses together — but each round raised the cost of an attack from an afternoon's work to a research project. Understanding the sequence is understanding how a hardened system is actually built, layer on grudging layer.

The classic bug: a buffer overflow becomes control

The archetypal memory-safety bug is the stack buffer overflow. A function allocates a fixed-size array on its stack frame and then copies attacker-controlled data into it without checking the length (\texttt{strcpy}, \texttt{gets}, a hand-rolled loop). The copy runs past the end of the buffer and keeps writing — and on a downward-growing x86 stack, "past the end" means over the saved return address that sits just above the locals.

When the function executes \texttt{ret}, the CPU pops that (now attacker-chosen) address into the instruction pointer and jumps there. The attacker has achieved control-flow hijack: they decide what the CPU does next. In the 1990s they simply pointed it at their own shellcode, sitting in the same overflowed buffer. Aleph One's 1996 "Smashing the Stack for Fun and Profit" turned this into a recipe a teenager could follow, and it defined the next thirty years of both attack and defence.

Its close cousin, the use-after-free (UAF), is today's dominant bug class in kernels and browsers alike. A chunk of memory is \texttt{free}d, the allocator hands the same address back for a different object, and a stale pointer to the old object is then used — letting an attacker read or write a controlled object through it. UAFs are so prevalent that they are the main reason the industry is migrating security-critical code to memory-safe languages like Rust.

Defence 1 — the stack canary

The first cheap defence: place a secret random value — a canary, after the coal-mine bird — between the local buffers and the saved return address when the function is entered. A contiguous overflow that reaches the return address must pass through the canary and corrupt it. The function's epilogue re-reads the canary before \texttt{ret}; if it changed, the program calls \texttt{\_\_stack\_chk\_fail} and dies instead of returning into the attacker's hands. Watch the smash happen.

GCC/Clang emit this automatically under \texttt{-fstack-protector-strong}. The canary is chosen once per process at start-up (from the kernel's random pool) and stored where the attacker cannot easily read it, so they cannot forge it. The bypass, of course, arrived fast: an information leak that discloses the canary lets the attacker write it back verbatim; and canaries do nothing against overflows that skip over them (indexed writes) or against use-after-free. So canaries raise the bar — they do not close the door.

Defence 2 — W^X (NX / DEP): make data non-executable

The 1990s exploit ran shellcode inside the stack buffer. So why is the stack executable at all? It need not be. W^X ("write XOR execute", also called NX / DEP) is the page-table policy that no page is ever both writable and executable at once: code pages are read-execute, data pages (stack, heap) are read-write-no-execute. The MMU's NX bit enforces it in hardware. Injected shellcode on the stack now faults the instant the CPU tries to execute it.

The bypass was ingenious and is now the standard technique: return-oriented programming (ROP). If you cannot inject new code, reuse the code already present. The attacker's overflow writes not one return address but a whole chain of them, each pointing at a short "gadget" — a handful of existing instructions ending in \texttt{ret} — already sitting in libc or the kernel. Stitched together, gadgets form a Turing-complete program built entirely from the victim's own bytes. W^X made code injection impossible and code reuse the new normal.

Defence 3 — ASLR / KASLR: hide the targets

ROP needs to know the addresses of its gadgets. Address-space layout randomisation (ASLR) attacks that assumption: at every execution the loader places the stack, heap, libraries, and (with PIE) the executable itself at random offsets. The attacker's pre-computed gadget addresses now point at garbage. KASLR is the same idea for the kernel — the kernel image and its modules load at a randomised base, so a userspace attacker who finds a kernel bug still does not know where the kernel is.

Randomisation's strength is measured in bits of entropy: if the base can land in one of 2^{b} aligned slots, the attacker faces b bits and, blind, must expect on the order of 2^{\,b-1} guesses. On 64-bit Linux userland ASLR typically gives b \approx 28–30 bits for libraries; KASLR historically gave a stingier \approx 9 bits (randomising the kernel base over a 1 GB window in 2 MB steps — 2^{30}/2^{21} = 2^{9}).

The bypass: entropy only helps if the layout stays secret. Any information-leak bug that discloses a single real pointer defeats the whole scheme — subtract the known offset and the entire map is recovered. Low-entropy KASLR was also brute-forceable, and 32-bit ASLR (only \approx 8 bits) was famously broken in minutes. Hence the modern refrain: ASLR is only as strong as your defence against info leaks — which is exactly why the side-channel speculative-execution attacks were so devastating: they leak the very addresses ASLR tried to hide.

Defence 4 — SMEP / SMAP: wall off user memory from the kernel

A whole family of kernel exploits used a shortcut: trick the kernel, while running in ring 0, into jumping to or reading from a page the attacker controls in user space (a "ret2usr" attack). The user page is trivially populated with a fake structure or shellcode, and the kernel — same address space, full privilege — happily uses it. Two CPU features slam this shut:

• SMEP (Supervisor Mode Execution Prevention): the kernel may not execute instructions from any user-mode page. A kernel control-flow hijack that redirects to userland shellcode now faults.
• SMAP (Supervisor Mode Access Prevention): the kernel may not even read or write user pages except in tightly-bracketed windows (Linux's \texttt{copy\_from\_user} toggles the \texttt{AC} flag via \texttt{stac}/\texttt{clac}). This kills attacks that plant a fake structure in user memory for the kernel to dereference.

Together SMEP+SMAP force kernel exploits to keep everything — code and data — inside kernel memory, which is far harder to control and, thanks to KASLR, harder to locate. The arms race responded: exploits pivoted to pure kernel-space ROP/JOP and to data-only attacks that never redirect control at all.

Defence 5 — CFI and the current frontier

ROP and its jump-oriented cousin (JOP) both violate the program's intended control-flow graph: they return to the middle of functions and call through pointers the compiler never sanctioned. Control-Flow Integrity (CFI) enforces the graph directly. Forward-edge CFI checks that every indirect call lands on a valid function entry of the right type; backward-edge protection (Intel CET's hardware shadow stack, or ARM's PAC pointer authentication) guarantees a \texttt{ret} can only go back to a real, recorded call site — which structurally defeats ROP. The kernel builds with Clang's \texttt{kCFI} today.

Rounding out the toolbox: stack-clash protection (a probe on every large stack growth so an allocation cannot silently leap the guard page into the heap, after the 2017 Stack Clash disclosures); heap hardening (safe-unlinking, randomised allocators, quarantines against UAF); and \texttt{-D\_FORTIFY\_SOURCE}, which swaps unbounded libc calls for length-checked variants at compile time. None is a silver bullet. The point of defence in depth is arithmetic: if each layer independently has a 10\% chance of being bypassable, chaining through four of them is a 0.1^4 = 0.0001 proposition.

The mitigations, in code

Two of these are pure arithmetic you can feel. First a canary check: a "clean" write leaves the canary intact and the function returns safely; an overflowing write clobbers it and the epilogue aborts. Second, ASLR entropy: how many guesses does randomisation buy? Run it and change the numbers.