Message Queues and Pub/Sub

It is Friday, 20:00, and a popular show just dropped. In sixty seconds, two million people press "buy." Your checkout service can comfortably charge fifty cards a second. If every "buy" button reached straight into the payment code — a synchronous request waiting for a reply — the payment service would be buried under forty thousand simultaneous calls, tip over, and take the whole shop down with it. Yet the shop stays up, every order eventually charges, and nobody notices. How?

The trick is to stop making the two services talk directly. Instead, the web servers drop each order into a queue — a buffer in the middle — and reply "got it!" instantly. The payment service then pulls orders out of the queue at its own steady pace of fifty a second. The queue absorbs the flood; the spike becomes a backlog that drains over the next few minutes. This is asynchronous messaging, and the humble queue in the middle is one of the most important structures in all of distributed systems.

Synchronous vs asynchronous, and what the queue buys you

In a synchronous call, the caller sends a request and blocks — it waits, doing nothing useful, until the reply comes back. The two parties are coupled tightly: they must both be alive at the same instant, and the caller runs no faster than the callee. That is exactly the socket/RPC model, and it is perfect when you need an answer right now (what's my balance?).

In asynchronous messaging, the sender (the producer) hands a message to a broker and moves on immediately, without waiting for it to be processed. Later — milliseconds or minutes later — a consumer picks the message up and acts on it. The queue between them is a simple idea with three superpowers:

Decoupling in time. Producer and consumer need not be alive at the same moment. The payment service can be redeployed, or crash and restart, and orders simply wait safely in the queue until it comes back. Nothing is lost.
Decoupling in rate — load levelling. A bursty producer and a steady consumer no longer have to match speeds. The queue soaks up the peaks and hands work out smoothly, so the consumer is sized for the average load, not the terrifying peak.
Decoupling in knowledge. The producer doesn't know or care who consumes the message, how many consumers there are, or what they do with it. You can add a second consumer (say, a fraud checker) without touching the producer at all.

The picture below is load levelling itself. The bursty curve is the arrival rate — a quiet baseline with a big spike when the show drops. The flat line is what a single consumer can actually process per second (drag the slider to change its capacity). While arrivals sit above the line, the gap is not lost — it piles up in the queue as backlog; when arrivals fall below the line, the consumer eats through that backlog. The queue turns a spike you cannot handle into a delay you can.

Two shapes: point-to-point queues and publish/subscribe

There are two fundamentally different messaging patterns, and choosing the wrong one is a common design error. They differ in a single question: when a message is delivered, does exactly one recipient get it, or do many?

Point-to-point (a work queue). Each message represents a task to be done once — charge this card, resize this image, send this email. Many consumers may be attached to the queue, but each message goes to exactly one of them. This is the competing consumers pattern: several identical workers pull from the same queue, and the broker hands each message to whichever worker is free. Add more workers and throughput rises — the queue automatically load balances across them. The task must not be done twice, so exactly one consumer wins each message.
Publish/subscribe (a topic). Each message is an event announced to the world — "order 42 was placed." The producer publishes to a topic, and every subscriber gets its own copy. This is fan-out: the "order placed" event might go simultaneously to the warehouse (to pick the item), the email service (to send a receipt), and the analytics pipeline (to update a dashboard) — three independent subscribers, each receiving the same event, none aware of the others.

Delivery guarantees, acknowledgements, and the dead-letter queue

A message queue faces the same partial-failure problem as any network communication: a consumer might crash after pulling a message but before finishing the work. If the broker deleted the message the instant it was handed out, that task would vanish. So brokers use acknowledgements:

The broker delivers a message but keeps it, marked "in flight."
Only when the consumer sends an ack ("done, safely processed") does the broker delete it.
If the consumer crashes or the ack doesn't arrive within a timeout, the broker redelivers the message to another consumer.

This gives at-least-once delivery: no message is ever lost, but a message whose ack was lost (not the message — the ack) gets processed twice. Just as with RPC retries, the consequence is unavoidable: consumers must be idempotent, because duplicates will happen. A message the broker gives up on after many failed attempts — malformed, or it keeps crashing its consumer — is moved aside to a dead-letter queue (DLQ), a holding pen for "poison" messages so one bad message can't block the whole queue forever. A human (or an alert) inspects the DLQ later.

You will see "exactly-once" on marketing pages, and it is almost a fib. True exactly-once delivery over an unreliable network is impossible — the ack can always be the thing that gets lost, forcing a redelivery. What systems like Kafka actually provide is exactly-once processing: at-least-once delivery, plus deduplication (each message carries an ID the consumer records) or transactional writes that make a duplicate a no-op. The effect happens once even though the message may be delivered twice. It is the same trick as RPC's exactly-once effect — and the same reason your consumer still has to be idempotent underneath.

Ordering, partitioning, and two flavours of broker

Do messages arrive in the order they were sent? Often you want them to — "account created" before "account updated." But strict global ordering fights against scale: if every message must be processed in one sequence, you can only have one consumer working at a time, which kills the parallelism that competing consumers gave you. The standard resolution is partitioning: split the stream into independent partitions by a key (say, the customer ID). Order is guaranteed within a partition but not across partitions — so all of one customer's events stay in order, while different customers are processed in parallel. Ordering and throughput are traded off by the granularity of the key.

Two influential broker designs sit at different points on this landscape:

RabbitMQ (a classic message broker, speaks AMQP). Think of it as a smart post office: it routes each message to queues, tracks per-message acks, and deletes a message once it is acknowledged. Rich routing, great for task queues and request/response-style work. The message is gone once consumed.
Kafka (a partitioned, append-only log). Think of it as a durable, replayable diary. Producers append events to the end of a partitioned log; consumers read forward by tracking their own offset (their bookmark). Messages are not deleted when read — they are kept for days, so a new consumer can replay history from the start, and many independent consumer groups can read the same log at their own pace. This makes Kafka the backbone of event-streaming and analytics pipelines, where "who has read this and can we delete it" is deliberately decoupled from "store the events."

Watch the backlog drain

Let's make load levelling and competing consumers concrete. Below, a producer emits a big burst of messages up front and then goes quiet, while consumers each process a fixed number per tick. We track the queue backlog every tick and watch it build, then drain — and we see how adding a second consumer halves the time to clear the flood.

// Arrivals per tick: a burst at the start (30, 25, 20) then a quiet trickle of 2/tick. const arrivals = [30, 25, 20, 2, 2, 2, 2, 2, 2, 2, 2, 2]; function simulate(consumers: number, ratePerConsumer: number): void { const capacity = consumers * ratePerConsumer; // total processed per tick let backlog = 0; let processed = 0; const bars: string[] = []; arrivals.forEach((incoming, tick) => { backlog += incoming; // producer adds work const done = Math.min(backlog, capacity); // consumers drain what they can backlog -= done; processed += done; bars.push(`t${tick}: ${"#".repeat(backlog).padEnd(30)} backlog=${backlog}`); }); console.log(`--- ${consumers} consumer(s), capacity ${capacity}/tick ---`); bars.forEach((b) => console.log(b)); const totalIn = arrivals.reduce((s, x) => s + x, 0); console.log(`total arrived=${totalIn}, total processed=${processed}\n`); } // One consumer: the burst piles up and drains slowly. simulate(1, 10); // Two competing consumers: double the drain rate, backlog clears much faster. simulate(2, 10);

With one consumer the backlog towers up to the 30s and takes many ticks to melt away; with two competing consumers the same flood clears in roughly half the time. The producer never changed — we just widened the outlet. That is the entire economic argument for competing consumers.

The queue's magic tempts a dangerous belief: "just put a queue in front of it and we can handle any load." No. A queue smooths a temporary burst on the assumption that, on average, the consumer keeps up. If arrivals exceed the consumer's capacity for a sustained period, the backlog does not smooth — it grows without bound. Two things then blow up: latency (a message added to a million-deep queue waits behind all million, so end-to-end delay climbs into minutes or hours) and memory/disk (the broker must store the ever-growing backlog until it runs out and starts dropping or rejecting messages). A queue converts an overload into delay, and delay is only survivable if the overload is brief. Monitor queue depth as a first-class health metric; if it trends up and never comes down, you have a capacity problem no queue can hide. And remember the other half: because delivery is at-least-once, the same message will occasionally be delivered twice — so every consumer must be idempotent. Processing "charge card" twice because an ack got lost is not a hypothetical; it is Tuesday.