Concurrent Servers

Your echo server works beautifully. You demo it, one client connects, types "hello", gets "hello" back. Then you launch it for real and the second visitor sits staring at a spinner — because your server is still deep inside a blocking read(), waiting on the first visitor, who wandered off to make tea. A server that talks to one client at a time is not a server; it is a queue of disappointed people. The entire art of server engineering is answering one question: how do you serve many clients at once?

Recall the shape of the socket API: accept() blocks until a client arrives, read() blocks until bytes come, write() can block when the send buffer is full. A single thread marching through accept → read → process → write for one client is, for that whole conversation, deaf to everyone else. This page surveys the classic ways out — process-per-connection, thread-per-connection, and worker pools — and the famous wall they all eventually hit: the C10K problem.

The trap: a serial accept-then-serve loop

Here is the naïve server, and it is worth seeing exactly why it fails to scale. It is not that it is slow per client — it is that it does clients strictly one after another.

// SERIAL server: correct, tiny, and hopelessly unscalable. const listener = listen(8080); while (true) { const conn = listener.accept(); // BLOCK until a client connects handle(conn); // ...then spend the WHOLE conversation here conn.close(); // only NOW do we loop back to accept the next one }

While handle(conn) runs — reading a request, hitting a database, writing a reply — no other client can even be accepted, let alone served. If each conversation takes 100 ms, this server can handle at most 10 clients per second, no matter how many CPU cores sit idle. And a single slow or malicious client that connects and never sends can freeze it forever. The fix in every case is the same idea: hand each accepted connection to its own unit of execution, and loop straight back to accept(). What that "unit of execution" is gives us the classic models.

The classic concurrency models

These four designs are the canon. Each trades differently between isolation, memory cost, and complexity — and the choice hinges on your understanding of threads vs processes.

Process per connection (fork). On each accept(), fork() a whole new process to handle that client. Strength: total isolation — a crash or memory corruption in one connection cannot touch another, and there are no shared-memory data races. This is the classic Apache prefork / inetd model. Weakness: a process is heavy — its own address space, its own file descriptors, and expensive context switches. Thousands of them exhaust memory fast.
Thread per connection. Same shape, but spawn a thread instead of a process. Strength: far lighter than a process — threads share the one address space, so they start faster and use less memory, and they can share caches and state directly. Weakness: that shared memory is a loaded gun — every shared structure now needs locks, and you inherit the full menagerie of race conditions and deadlocks. And threads, though lighter than processes, still are not free.
Thread/worker pool. Instead of spawning per connection, start a fixed pool of N worker threads at boot. The accept() loop drops each new socket onto a shared queue; idle workers pull from it. Strength: bounded resources — you never have more than N threads no matter how many clients arrive, so memory and scheduling stay predictable; a load spike grows the queue, not the thread count. This is the workhorse design for most application servers. Weakness: if all N workers are stuck on slow work, new connections wait in the queue; you must size the pool to the workload.
Pre-forking. A hybrid: fork a pool of processes up front, and let them all accept() on the shared listening socket (the kernel hands each connection to one of them). You get process isolation and a bounded, pre-warmed pool with no per-connection fork cost. This is how production Apache/nginx worker models and many WSGI servers (Gunicorn) run.

The wall: the C10K problem

"Just spawn a thread per connection" feels like it should scale forever — but around ten thousand concurrent connections (the famous "C10K" figure, coined by Dan Kegel in 1999) the thread-per-connection model falls apart. The reasons are structural, not a matter of buying a bigger machine:

Memory. Each thread reserves a stack — often ~1 MB by default. Ten thousand threads is ~10 GB of stacks before you have processed a single byte of actual data.
Context-switch cost. The OS scheduler must time-slice among all runnable threads. The more threads, the more time the CPU spends switching between them (saving/restoring registers, flushing caches, walking the scheduler's run queue) rather than doing useful work.
Scheduler pressure. Most of those ten thousand threads are blocked waiting on I/O at any instant. Keeping ten thousand mostly-idle threads alive just to represent "connections that might have data soon" is enormously wasteful — you are paying full thread price for connections that are doing nothing.

The counter-intuitive punchline: past a point, adding threads makes throughput go down. The chart contrasts the fantasy (throughput scales linearly with concurrency) against the reality (throughput rises, plateaus, then falls as overhead dominates). Drag toward the right and watch the realistic curve turn over.

The realistic curve is the Universal Scalability Law in action: useful throughput T(n) = \dfrac{n}{1 + \alpha(n-1) + \beta\, n(n-1)} where the \alpha term is serialisation (contention on shared locks) and the \beta term is crosstalk — the coordination and context-switch cost that grows with the square of the thread count, and eventually drags the whole curve back down.

Modelling a bounded worker pool

The pool design exists precisely to sit near the top of that curve rather than sliding down its right-hand slope. Let's model it. Suppose each request needs a base amount of work, but running many workers concurrently inflates every worker's effective time by a small per-worker context-switch overhead. Then throughput is workers / effective-time-per-job — and there is a sweet spot beyond which more workers are self-defeating.

// Universal Scalability Law: contention (alpha) and crosstalk (beta) overhead. // alpha serialises shared work; beta is context-switch cost, growing with n^2. const ALPHA = 0.03, BETA = 0.006; // Throughput relative to one worker. Ideal would be exactly n (linear speedup). function throughput(n) { return n / (1 + ALPHA * (n - 1) + BETA * n * (n - 1)); } console.log("workers | throughput | vs ideal (linear n)"); console.log("--------+------------+--------------------"); let best = { workers: 0, tp: 0 }; for (const w of [1, 2, 4, 8, 16, 32, 64, 128, 256]) { const tp = throughput(w); const pct = ((tp / w) * 100).toFixed(0); // ideal linear speedup is exactly w console.log(` ${String(w).padStart(4)} | ${tp.toFixed(2)} | ${pct}% of ideal`); if (tp > best.tp) best = { workers: w, tp }; } console.log(`\nPeak throughput near ${best.workers} workers (${best.tp.toFixed(2)}).`); console.log("Past the peak, every doubling ADDS threads while REMOVING throughput —"); console.log("crosstalk (the n^2 term) takes over. That is the C10K wall.");

Notice the efficiency column collapsing: at 256 workers you are running 256 threads to do the work of a handful, because each one now spends most of its life being switched in and out. A bounded pool sized to the peak gives you the most work for the least resource — and a load spike simply lengthens the queue, which is a far cheaper thing to grow than the thread count.

The seductive intuition is linear: "twice the threads, twice the work." It is wrong, and believing it is how servers fall over under load. Past the sweet spot, every extra thread adds memory pressure and context-switch overhead while subtracting from the CPU time available for real work — so throughput peaks and then declines. Spawning a thread per connection and hoping to reach ten thousand of them is the textbook way to hit this wall (C10K). The cures are bound the concurrency (a fixed worker pool with a queue) and, when even that isn't enough, stop using a thread-per-connection at all — serve thousands of connections from a single thread with non-blocking I/O and an event loop, the subject of the next page.

Two escapes. The first, above, is to abandon "one thread per connection": an event loop keeps just a handful of threads and multiplexes tens of thousands of sockets over them, paying almost nothing for an idle connection. The second is to make the threads themselves radically cheaper: green threads / goroutines / async tasks (Go, Erlang, Rust/async, Java's Project Loom) are user-space "threads" that cost a few kilobytes, not a megabyte, and are scheduled by a runtime onto a small pool of real OS threads. Both answers reject the premise that a connection needs a whole kernel thread to wait on it — which, once you have met the event loop, feels obvious.