You have a program that runs beautifully on 4 cores. You move it to a 64-core server expecting a 16\times boost — and it runs slower than on 4. Not a little slower; measurably, humiliatingly slower. The culprit is almost always a single hot lock. This lesson is about the physics of that collapse: why one lock does not merely fail to help but actively destroys throughput as cores pile on, and the toolkit — per-CPU data, sharding, RCU, cache-line padding — that kernel engineers use to claw scalability back.

The intuition from Amdahl's law already warns you that a serial fraction caps speedup. But Amdahl is optimistic: it assumes the serial part costs the same no matter how many cores wait. Real locks are worse, because contention adds a cost that grows with the number of contenders — the coherence traffic of shuttling the lock's cache line between them. Past a point, adding a core subtracts throughput.

Two costs, not one

A single global lock imposes two distinct penalties as you add cores, and it is vital to keep them apart:

• Serialization — only one core can be inside the critical section at a time, so that work cannot go in parallel. This is the Amdahl term. It flattens throughput to a ceiling but does not, by itself, make things go backwards.
• Coherence / crosstalk — every core that tries the lock must pull its cache line to its own cache in Modified state, invalidating everyone else. This traffic grows roughly as the square of the number of contenders, and it is pure overhead stolen from useful work. This is the term that turns the curve downward.

The Universal Scalability Law (USL) captures both. If N cores would ideally give throughput N, the real throughput is

Here \alpha is the contention (serialization) coefficient and \beta is the coherence (crosstalk) coefficient. With \beta = 0 you recover an Amdahl-like ceiling. But any \beta > 0 gives the curve a peak: it rises, tops out, and then declines — the retrograde scalability that defines a contended lock.

Watch the curve turn over

Below is the USL. The dashed line is the fantasy — linear scaling, throughput equal to core count. The solid curve is reality. Drag \alpha (contention) and watch the ceiling drop; then nudge \beta (coherence crosstalk) up off zero and watch the curve stop being a ceiling and become a hill — throughput peaks and then falls off a cliff as cores are added. That downturn is the single global lock destroying your machine. The peak sits at N^\star = \sqrt{(1-\alpha)/\beta}; adding cores past it is worse than useless.

The toolkit: how kernels claw scalability back

Every scalable-synchronization technique is, at heart, a way to make cores stop touching the same cache line. If they don't share the line, there is no coherence traffic, and \beta collapses toward zero. The kernel toolkit:

• Per-CPU data — give each core its own private copy of the counter/statistic/free-list, so the common path touches only local memory (no sharing at all). Aggregate lazily when someone actually needs the global total. This is why kernel statistics counters are per-CPU.
• Sharding / lock striping — replace one lock over a big table with k locks, each over a slice, hashing keys to shards. Contention drops by a factor of k because independent operations touch independent locks (Java's old \texttt{ConcurrentHashMap} did exactly this).
• RCU — for read-mostly data, remove the read-side lock entirely (see RCU); readers touch no shared writable line at all.
• Cache-line padding — eliminate false sharing (below) by padding hot fields to their own cache line.

False sharing: contention you never asked for

The cruellest scalability bug is false sharing. Two cores update two completely independent variables — no logical contention whatsoever — but the two variables happen to sit in the same 64-byte cache line. Coherence works at cache-line granularity, so each write invalidates the whole line in the other core's cache, and the line ping-pongs exactly as if the cores were fighting over one lock. Your carefully lock-free per-core counters silently serialize because the compiler packed them adjacently. The demo shows the cost.

Reader-writer locks are not a free lunch

A tempting "fix" for a read-heavy lock is a reader-writer lock: many readers concurrently, writers exclusive. But it disappoints on many cores for a subtle reason: readers still write. To register as a reader, each must atomically bump a shared reader count — and that shared counter's cache line ping-pongs between all the readers exactly like a plain lock. So a rwlock under heavy read load can scale no better than a mutex, because the "read lock" itself is a contended write. This is precisely the gap RCU fills: RCU readers write nothing, so there is no shared line to bounce. The lesson: a lock that requires any shared writable state on the hot path cannot escape the coherence tax — the only true fix is to stop sharing the line.

This crystallises into a design principle, the scalability commutativity rule: whenever two operations commute — their results don't depend on the order they run — it is possible to implement them so they scale (require no communication). If operations are forced to share mutable state, look for a reformulation where they commute; if they genuinely don't commute, no amount of clever locking will make them scale, and you must change the interface, not the lock.