Service Discovery and Microservices

Once upon a time an application was one program — a single codebase, compiled into a single deployable lump, running as one process (a monolith). Inside it, the checkout code called the inventory code with a plain function call: instant, reliable, and either both ran or the whole thing crashed together. Then the lump got too big to reason about, too big for one team to deploy without stepping on another, and too coarse to scale (you couldn't add capacity to just the image resizer without cloning the entire application). So people broke the monolith apart into microservices: many small, independently-deployed services, each owning one job, each talking to the others over the network.

That last phrase is the whole story. The instant "checkout calls inventory" stops being a function call and becomes a network call, everything you learned on the earlier pages of this course arrives at once — RPC's leaky abstraction, load balancing, partial failure, timeouts. This page is about the two problems that architecture creates and the patterns that tame them: how services find each other when they come and go, and how they survive each other's failures.

The microservices tradeoff

Microservices are not automatically "better" than a monolith — they trade one set of problems for another, and the trade is only worth it at a certain scale of team and system.

You gain	You pay
Independent deploys — ship one service without redeploying everything	A network hop between every call: latency, and calls that can now fail on their own
Independent scaling — add instances of just the hot service	Distributed failure — one slow service can stall the callers waiting on it
Team autonomy — each team owns and releases its own service	Operational complexity — dozens of services to deploy, monitor, and trace
Technology freedom — pick the right language/DB per service	Hard-won data consistency — no single database transaction spans services

The blunt summary: microservices trade a function call for a network call. A monolith is simpler and faster and should be the default for a small system; you reach for microservices when the organisational and scaling gains outweigh the very real cost of turning in-process certainty into distributed uncertainty.

The discovery problem

In a monolith, inventory's code is right there in memory. In microservices, the inventory service runs as some number of instances, on some machines, at some IP addresses — and those change constantly. Instances are created and destroyed as traffic rises and falls (autoscaling), moved when a host is drained for maintenance, restarted with a new IP after a crash. Hard-coding inventory = 10.2.3.4:8080 is hopeless: that address is stale by lunchtime. So: how does the checkout service find a healthy inventory instance to call, right now?

The answer is a service registry: a live, authoritative directory of "which services exist and where their healthy instances are." It works in three beats:

Registration. When an inventory instance starts, it announces itself to the registry: "I'm inventory, I'm at 10.2.3.4:8080." When it shuts down, it deregisters.
Health & TTL. The registry health-checks each instance (or the instance sends a periodic heartbeat). An instance that stops heartbeating within its TTL (time-to-live) is presumed dead and dropped from the directory automatically — so a crashed instance silently disappears rather than receiving doomed traffic.
Lookup. The checkout service asks the registry "give me a healthy inventory instance," gets back a current address, and calls it. Tools built for this include Consul, etcd, and ZooKeeper (the last two are consensus-backed, so the directory itself is consistent and fault-tolerant).

There are two places the lookup logic can live, and the distinction is worth knowing:

Client-side discovery. The caller queries the registry itself, gets the list of instances, and picks one (doing its own load balancing). Fewer moving parts, but every client needs the discovery logic baked in.
Server-side discovery. The caller just sends the request to a fixed load balancer / router, which consults the registry and forwards to a healthy instance. The client stays dumb; the routing tier does the work. This is what a Kubernetes Service does under the hood.

The API gateway

If a system has forty microservices, you do not want the outside world — mobile apps, browsers, partners — talking to forty different addresses and each learning forty sets of rules. Instead you put a single front door in front of them all: the API gateway. Every external request enters through the gateway, which then routes it to the right internal service. Because all traffic funnels through this one place, it is the natural home for the cross-cutting concerns you'd otherwise reimplement in every service:

Routing — map /orders/* to the order service, /users/* to the user service.
Authentication & authorization — check the caller's token once, at the edge, so internal services can trust it.
Rate limiting — throttle abusive callers before they reach the fragile internals.
TLS termination, request logging, response aggregation — done in one place, consistently.

The gateway is to a fleet of services what a receptionist is to an office building: one entrance, one set of rules, and the visitor never needs to know which floor anyone sits on.

Resilience: timeouts, retries, and the circuit breaker

Now the failures. Because every inter-service call is a network call, every one of them can be slow, fail, or hang forever. A resilient service defends itself with a small, layered toolkit:

Timeouts. Never wait forever. Cap every call, so a dead dependency turns into a prompt error you can handle instead of a hung thread that ties up resources.
Retries with backoff and jitter. A failed call might just be a transient blip, so retry — but not immediately and not in lockstep. Wait a bit and increase the wait each time (exponential backoff: 1s, 2s, 4s…), and add a random jitter so a thousand clients that all failed at the same instant don't all retry at the same instant.
The circuit breaker. The star of the show — a pattern that stops a struggling service from being hammered to death.

A circuit breaker wraps calls to a dependency and watches them, exactly like the electrical breaker in your home that trips to protect the wiring. It has three states:

Closed — normal. Calls pass through to the dependency; the breaker counts failures.
Open — tripped. Once failures cross a threshold, the breaker stops calling the dependency altogether and fails fast (returns an error, or a fallback, immediately). This gives the sick service breathing room to recover instead of being pounded, and stops callers wasting time on calls that will fail anyway.
Half-open — probing. After a cooldown, the breaker lets a single trial call through. If it succeeds, the dependency is healthy again and the breaker closes (normal service resumes); if it fails, the breaker re-opens and waits again.

Watch a circuit breaker trip and recover

Let's drive a breaker through a real incident: a downstream service goes down for a while, then recovers. We use a logical clock (ticks) instead of wall time. Watch it trip to OPEN, short-circuit the doomed calls (saving everyone the wait), probe once in HALF-OPEN, and close again when the trial call finally succeeds.

type State = "CLOSED" | "OPEN" | "HALF_OPEN"; class CircuitBreaker { state: State = "CLOSED"; private failures = 0; private openedAt = 0; constructor(private threshold: number, private cooldown: number) {} call(now: number, downstreamOk: boolean): string { // In OPEN, refuse to call until the cooldown elapses. if (this.state === "OPEN") { if (now - this.openedAt < this.cooldown) return "OPEN → short-circuit (fail fast)"; this.state = "HALF_OPEN"; // time to probe } if (this.state === "HALF_OPEN") { if (downstreamOk) { this.state = "CLOSED"; this.failures = 0; return "HALF-OPEN trial OK → CLOSED"; } this.state = "OPEN"; this.openedAt = now; return "HALF-OPEN trial FAILED → OPEN"; } // CLOSED: really make the call. if (downstreamOk) { this.failures = 0; return "call OK"; } this.failures++; if (this.failures >= this.threshold) { this.state = "OPEN"; this.openedAt = now; return `call FAILED (#${this.failures}) → TRIP OPEN`; } return `call FAILED (#${this.failures})`; } } // Downstream health per tick: healthy, then DOWN for ticks 2..8, then healthy again. const health = [true, true, false, false, false, false, false, false, false, true, true, true, true]; const cb = new CircuitBreaker(3, 3); // trip after 3 failures; probe after 3 ticks for (let now = 0; now < health.length; now++) { const result = cb.call(now, health[now]); console.log(`t${String(now).padStart(2)} down=${!health[now] ? "Y" : "."} [${cb.state.padEnd(9)}] ${result}`); }

Trace it: the first three failures trip the breaker OPEN; the next several calls are short-circuited — the breaker refuses to even attempt them, so the failing service isn't pounded and callers get an instant error instead of a slow timeout. After the cooldown it goes HALF-OPEN and probes; because the downstream is still down, the trial fails and it re-opens. Once the downstream recovers, the next probe succeeds and the breaker snaps back to CLOSED. That is fault isolation in thirty lines.

Microservices trade a function call for a network call, and that changes the risk arithmetic completely: every call can now be slow, fail, or time out, and the tempting reflex — "just retry it" — is how a small hiccup becomes a self-inflicted outage. Picture one service getting a bit slow. A thousand callers time out at once and all retry immediately; now the struggling service faces double the load, gets slower still, triggering more timeouts and more retries — a retry storm where your own fleet DDoSes the weakest link and drags the whole system down. This is why naive retries are dangerous and the defences are non-negotiable: retry with exponential backoff (wait longer each attempt) and jitter (randomise the wait, so clients don't synchronise into a thundering herd), cap the number of attempts, and wrap the dependency in a circuit breaker so that once it's clearly failing you stop calling it entirely and let it recover. Retrying without backoff, jitter, and a breaker doesn't add resilience — it adds a positive feedback loop that turns a blip into a collapse.