Caching and CDNs

Somewhere on the other side of the planet is the one true copy of this page's data — the origin server. If every reader had to reach all the way back to it for every image, script and byte, the web would crawl: a request from Sydney to a server in Virginia is a 300-millisecond round trip before the server does any work, and the origin would collapse under the sheer number of identical requests. Yet the web feels instant. The reason is that almost nothing you load actually comes from the origin. It comes from a cache — a nearer, faster copy — and there is a whole ladder of them between you and the truth.

A cache is just replication with a particular motive: not durability, but speed. Keep a copy of expensive-to-fetch data somewhere cheap-to-reach, and serve future requests from the copy. Simple — until you ask the question that makes caching genuinely hard: when the original changes, how does the copy find out? That question, cache invalidation, is the villain of this whole story, and we will build up to it.

Caches all the way down

The same data is copied at layer after layer, each nearer to the user and faster than the last. A single page load might be satisfied at any rung of this ladder:

Browser cache. The closest copy of all — on your own disk. A second visit to a site re-uses images and scripts it already downloaded, with zero network at all.
Forward / reverse proxy caches. A shared cache in front of many users (a corporate proxy) or in front of the origin (a reverse proxy like nginx or Varnish), serving one copy to everyone who asks.
CDN edge cache. A globe-spanning network of caches parked physically near users, so the copy is milliseconds away instead of an ocean away. The heart of this page — more below.
Application cache. Inside your own system, a fast in-memory store like Redis or memcached holds the results of expensive database queries and computations, so the app answers from RAM instead of re-querying the database.

Every rung trades a little staleness risk for a lot of speed, and the art of caching is choosing that trade deliberately at each layer.

How HTTP caching actually works

The web has a built-in caching protocol baked into HTTP headers. It runs on two ideas: an expiry (how long a copy may be used without checking) and a validator (a cheap way to ask "is my copy still good?" without re-downloading).

Header	Set by	What it does
`Cache-Control: max-age=3600`	server	"This copy is fresh for 3600 seconds — serve it without asking."
`Cache-Control: no-store`	server	"Never cache this at all" (e.g. a bank statement).
`Cache-Control: no-cache`	server	"Cache it, but always revalidate before using."
`Expires: <date>`	server	Older absolute-time form of an expiry (superseded by max-age).
`ETag: "v3-abc"`	server	An opaque fingerprint (version tag) of the resource's current content.
`If-None-Match: "v3-abc"`	client	"I have version v3-abc — only resend if it changed."
`304 Not Modified`	server	"Your copy is still good" — an empty, tiny reply; no re-download.

The lifecycle: the server sends a resource with max-age=3600 and an ETag. For the next hour the browser serves its copy with no network. After that the copy is stale — but stale does not mean wrong, it means "must check." So the browser sends a conditional request, If-None-Match: "v3-abc". If the content is unchanged, the server replies 304 Not Modified — a few bytes, no body — and the browser keeps using its copy for another hour. Only if the content genuinely changed does the server send the full new resource. This revalidation makes "check if fresh" almost free compared to "download again."

Eviction and hit ratio: a cache is finite

A cache is smaller than the data it fronts — that is the whole point, it lives in fast, scarce memory. So when it fills up and a new item must go in, something has to be thrown out. The rule for choosing the victim is the eviction policy:

LRU (Least Recently Used). Evict the item that has gone longest without being accessed. Bets that recently-used things will be used again soon (temporal locality) — the default choice, and usually a good one.
LFU (Least Frequently Used). Evict the item with the fewest accesses over time. Better for a workload with a few permanently-popular items and a churn of one-offs around them.

The score that matters is the hit ratio: the fraction of requests served from the cache rather than passed through to the origin. A hit ratio of 0.9 means the origin sees only one request in ten — a tenfold reduction in load, and most users getting the fast path. Hit ratio climbs with cache size, but with diminishing returns: once the cache holds the "hot" working set, making it bigger buys little, because the long tail of rarely-touched items would be evicted before they're ever reused anyway.

Drag the working-set slider: a workload with a small, concentrated hot set saturates its hit ratio with a tiny cache; a workload spread over a huge working set needs a far bigger cache to reach the same ratio. Right-sizing a cache is reading exactly this curve for your traffic.

Build an LRU cache and measure its hit ratio

Let's implement LRU from scratch and run a request stream through it. The trick is to keep items in recency order: on every access, move the item to the "most recent" end; when full, evict from the "least recent" end. A JavaScript Map remembers insertion order, so deleting and re-inserting a key on access is a neat way to keep the order honest.

class LRUCache { private cap: number; private store = new Map<string, number>(); hits = 0; misses = 0; constructor(cap: number) { this.cap = cap; } get(key: string): number | undefined { if (!this.store.has(key)) { this.misses++; return undefined; } this.hits++; const val = this.store.get(key)!; this.store.delete(key); // remove... this.store.set(key, val); // ...and re-insert → now most-recent return val; } put(key: string, val: number): void { if (this.store.has(key)) this.store.delete(key); this.store.set(key, val); if (this.store.size > this.cap) { const oldest = this.store.keys().next().value; // least-recently-used this.store.delete(oldest); console.log(` evicted "${oldest}"`); } } } // A request stream with locality: "a" and "b" are hot; c,d,e are one-offs. const stream = ["a","b","a","c","a","b","d","a","b","e","a","b","c","a","b"]; function run(capacity: number): void { const cache = new LRUCache(capacity); console.log(`--- LRU capacity ${capacity} ---`); for (const key of stream) { if (cache.get(key) === undefined) cache.put(key, 1); // miss → fetch + store } const total = cache.hits + cache.misses; const ratio = (cache.hits / total).toFixed(2); console.log(`hits=${cache.hits} misses=${cache.misses} hit ratio=${ratio}\n`); } run(2); // too small: the hot pair keeps fighting the one-offs for space run(4); // roomy enough to hold the hot set: far better hit ratio

With capacity 2 the cache thrashes — every one-off request (c, d, e) evicts part of the hot set, which then misses next time. Bump the capacity to 4 and the hot items a and b stay resident, so the hit ratio jumps. Same stream, same policy — only the size changed. That is the diminishing-returns curve above, measured.

Content Delivery Networks

A CDN is caching turned into planet-scale infrastructure: a company (Cloudflare, Akamai, Fastly) runs thousands of cache servers — edge or Points of Presence — in data centres in hundreds of cities. Your site's static assets (and increasingly its dynamic responses) are cached at these edges, close to wherever your users physically are. A reader in Sydney is served by an edge in Sydney; a reader in Oslo by an edge in Oslo. The 300-millisecond ocean crossing becomes a 5-millisecond hop across town.

How does a user's request reach the nearby edge instead of the distant origin? By the same steering tricks a load balancer uses: DNS hands each user the address of a close edge, or — more cleverly — the CDN uses anycast, announcing the same IP address from every edge so the internet's own routing delivers each user to the topologically nearest one. When an edge gets a request for something it hasn't cached (a miss), it fetches from the origin once, caches it, and serves everyone else from the copy. To spare the origin from thousands of edges all missing at once, CDNs use origin shielding: a designated middle tier that the edges pull through, so the origin sees a trickle of requests instead of a flood.

Here is a failure that catches everyone once. A wildly popular item is cached with a TTL, and at the instant that TTL expires, the copy vanishes from the cache. In that same instant, ten thousand concurrent requests for it all find a miss — and all ten thousand simultaneously charge the origin database to recompute the very same value. The origin, sized to handle the trickle of misses, is flattened by the stampede; it slows, requests pile up, and the site falls over precisely because something was popular. This is the thundering herd or cache stampede. The cures: let only the first requester recompute while the others wait for it (request coalescing / a lock); serve the slightly-stale value while one background worker refreshes it (stale-while-revalidate); or jitter the TTLs so a million keys don't all expire at the same second. The common theme — never let a cache miss turn into an uncoordinated mob.

The hard part: invalidation and staleness

Everything so far has a cost hiding in it. Every cached copy is a bet that the original hasn't changed — and sometimes you lose that bet. A copy the cache still believes is fresh, but which no longer matches the origin, is stale, and serving it means serving wrong data: an old price, a deleted post, last week's headline. Fixing this is cache invalidation — telling every copy, everywhere, "throw yours away, it's out of date" — and it is genuinely hard because the copies are many, distributed, and not all reachable at once.

There is no free lunch here; it is a CAP-flavoured tradeoff between freshness and speed/load. The main strategies:

TTL (time-to-live) — expiry. Let each copy live for a fixed time, then expire. Dead simple and needs no coordination, but you accept staleness up to one TTL. Short TTL = fresher but more origin load; long TTL = less load but staler. Choosing the TTL is choosing your staleness.
Explicit purge. When the data changes, actively tell the caches to drop that key (a CDN "purge" API call, an event that busts the app cache). Fresh almost immediately, but you must reliably reach every cache, and it's easy to forget a layer.
Versioned keys (cache busting). Never invalidate — change the name. Serve app.9f3c1.js instead of app.js; when the file changes, the hash in the name changes, so it's a brand-new key that was never cached. The old key simply ages out. This sidesteps invalidation entirely and is why build tools fingerprint asset filenames.

Phil Karlton's famous quip — "There are only two hard things in computer science: cache invalidation and naming things" — is a joke that is also completely true. Caching is easy to add and feels like free speed, which is exactly the trap: the moment you cache something, you have created a second copy that can silently disagree with the first, and now every write to the original must somehow reckon with every copy of it scattered across browsers, proxies, and edges worldwide. A stale cache does not error; it confidently serves the wrong answer — an old balance, a just-deleted comment, a price you no longer honour — and those bugs are maddening precisely because the code "works." The discipline is to treat every cache entry's lifetime as a deliberate decision: pick a TTL you can defend, know exactly how a change propagates (purge? version? expiry?), and never cache data whose staleness you cannot tolerate (use no-store for the bank statement). Caching is a promise about time, and you must decide, on purpose, how much time you can afford to be wrong.