RPC and gRPC

Here is a function call you have written a thousand times:

const total = getAccountBalance("alice");

The machine pushes an argument, jumps to the function, runs it, and hands back a value. Microseconds, deterministic, boring. Now imagine that getAccountBalance does not live in your program at all — it runs on a server three thousand kilometres away, guarding a database you cannot even see. What if you could call it with exactly the same line of code, and have the argument fly across the internet, the function run over there, and the answer come back — all hidden behind that innocent getAccountBalance("alice")?

That dream is Remote Procedure Call (RPC): make a call to another machine look and feel like a local call. It is one of the oldest and most seductive ideas in distributed computing — and, as we will see, one of the most quietly dangerous, because the network it hides never truly goes away. This page builds on the idea of a remote procedure call toward the machinery of a modern RPC system, and then makes it concrete with gRPC, the framework running inside a large fraction of today's data centres.

Stubs, skeletons, and marshalling

For a remote call to look local, something must stand in for the far-away function on each side. Those stand-ins are generated code, and they have names worth memorising:

The client stub is a fake local function with the same signature as the real one. You call getAccountBalance("alice"); secretly the stub packs the method name and arguments into a byte buffer, ships it over the network, waits, unpacks the reply, and returns it — so to you it was just a function that returned a number.
The server skeleton (or dispatcher) does the mirror image: it receives the buffer, unpacks the method name and arguments, calls the real function, packs the return value back up, and sends it home.

The packing and unpacking is marshalling (and unmarshalling): turning in-memory values — integers, strings, structs, lists — into a flat sequence of bytes that can travel a wire, and back again. It is the same problem as serialization and framing, applied to whole call: which method, with which arguments, returning what. Marshalling has to agree on byte order, how integers are encoded, how a string's length is recorded, how a missing field is represented — because the two machines may run different CPUs, languages, and operating systems.

The beautiful part: you write none of this. Both stubs are generated automatically from a single description of the interface — which is the next piece.

The IDL: one contract, many languages

How do the client stub and the server skeleton agree on exactly what a call looks like — the method names, the argument types, the return types — when the client might be written in TypeScript and the server in Go? They share a single, language-neutral description called an Interface Definition Language (IDL). You write the interface once; a code generator emits the stub in the client's language and the skeleton in the server's language, both guaranteed to marshal bytes the same way.

The IDL is the contract. Here is a tiny one in gRPC's IDL, Protocol Buffers:

// accounts.proto — the shared contract syntax = "proto3"; message BalanceRequest { string account_id = 1; } message BalanceReply { int64 cents = 1; } service Accounts { rpc GetBalance(BalanceRequest) returns (BalanceReply); }

From this one file, protoc generates a GetBalance stub for the client and a GetBalance handler interface for the server, in whatever languages you ask for. The odd little numbers (= 1) are field tags — the field's identity on the wire. The name account_id is for humans; the tag 1 is what actually gets written into the bytes. That indirection is what lets Protocol Buffers evolve a schema without breaking old clients, which we will return to in a moment.

Watch the marshalling happen

Let's strip a stub down to its essence and actually run it. Below, a toy stub marshals a call into a request "object" (a simple tagged string standing in for the wire bytes), a toy skeleton unmarshals it, runs the real function, and marshals a reply that the stub unmarshals back into a plain return value. No real network — but every step a real RPC framework performs is here.

// ---- the REAL functions that live "on the server" ---- const impl: Record<string, (args: string[]) => string> = { getBalance: (args) => (args[0] === "alice" ? "25000" : "0"), add: (args) => String(Number(args[0]) + Number(args[1])), }; // ---- MARSHAL: pack a method name + args into a flat "wire" string ---- function marshal(method: string, args: string[]): string { // frame = method | argCount | each arg length-prefixed (see serialization/framing) const parts = [method, String(args.length), ...args.map((a) => a.length + ":" + a)]; return parts.join("|"); } function unmarshal(wire: string): { method: string; args: string[] } { const [method, n, ...rest] = wire.split("|"); const args = rest.slice(0, Number(n)).map((p) => p.slice(p.indexOf(":") + 1)); return { method, args }; } // ---- the SERVER SKELETON: unmarshal, dispatch to impl, marshal reply ---- function serverHandle(requestBytes: string): string { const { method, args } = unmarshal(requestBytes); const result = impl[method](args); return "OK|" + result; // marshal the reply } // ---- the CLIENT STUB: looks like a normal function, hides all of the above ---- function callRemote(method: string, ...args: string[]): string { const requestBytes = marshal(method, args); console.log(` → on the wire: "${requestBytes}"`); const replyBytes = serverHandle(requestBytes); // "network round trip" console.log(` ← reply bytes: "${replyBytes}"`); return replyBytes.split("|")[1]; // unmarshal → plain value } // ---- your code just calls it like a local function ---- console.log("getBalance('alice'):"); console.log("returned:", callRemote("getBalance", "alice")); console.log("\nadd(20, 22):"); console.log("returned:", callRemote("add", "20", "22"));

Notice how callRemote(...) reads like an ordinary call and hands back an ordinary value. That is the whole magic trick — and also, as the next section argues, the whole trap.

The leaky abstraction: a remote call is not a local call

Making a remote call look local is a wonderful convenience and a subtle lie. A local call and a remote call differ in ways no amount of generated code can paper over:

Latency. A local call is nanoseconds; a call across a data centre is hundreds of microseconds; across an ocean, hundreds of milliseconds — a factor of a million. Code that calls getBalance in a tight loop is fine locally and a catastrophe remotely.
Partial failure. A local call either runs or your whole process crashes. A remote call has a terrifying third outcome: the request left, but you never hear back. Did the server run it and the reply got lost? Did the request never arrive? You genuinely cannot tell from the caller's side — and that ambiguity is the source of most distributed-systems pain.
No shared memory. Locally you can pass a pointer to a huge object for free. Remotely there are no shared pointers — everything must be copied (marshalled) across, so a "reference" argument silently becomes a deep copy, and mutating it does not affect the caller's copy.
Concurrency and ordering. Other clients are calling the same server at the same time; messages can arrive out of order; the network can duplicate them. None of this exists for a local call.

These are not implementation bugs to be fixed; they are the nature of a network. The classic catalogue of the assumptions that RPC tempts you to forget is the eight fallacies of distributed computing (L. Peter Deutsch and colleagues at Sun): the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn't change; there is one administrator; transport cost is zero; the network is homogeneous. Every one is false, and every RPC that pretends otherwise eventually pays for it.

gRPC, concretely

gRPC (open-sourced by Google in 2015) is a modern RPC framework that makes three specific, opinionated choices for the three pieces we have met — the IDL, the wire format, and the transport:

IDL and wire format: Protocol Buffers. The same .proto file is both the interface contract and the marshalling scheme. Protobuf is a compact binary format — it writes field tags and values, not field names, so a message is far smaller than the equivalent JSON, and parsing is fast. Because fields are identified by tag number, you can add new fields without breaking old readers (they skip tags they don't recognise) — schema evolution built in.
Transport: HTTP/2. gRPC runs over HTTP/2, which multiplexes many concurrent calls over one TCP connection (no head-of-line blocking between calls), supports binary framing and header compression, and — crucially — allows streaming in both directions on a single call.
Four kinds of call. Because HTTP/2 streams flow both ways, gRPC is not limited to "one request, one reply." It offers four call shapes.

Call type	Client sends	Server sends	Good for
Unary	one message	one message	ordinary request/reply (getBalance)
Server-streaming	one message	a stream of messages	subscribe to updates, download a large result set
Client-streaming	a stream of messages	one message	upload a file in chunks, batch of metrics
Bidirectional	a stream	a stream	chat, live telemetry, a two-way conversation

JSON is lovely to read and debug, and REST-over-JSON dominates public web APIs for exactly that reason. But inside a data centre, where two services exchange millions of messages a second, JSON's costs bite: it ships the field names as text on every message, numbers are decimal strings that must be parsed, and there is no schema to catch a typo'd field. Protobuf ships only tiny numeric tags and packed binary values, is typically several times smaller, and parses much faster — and the .proto schema is checked at compile time. The tradeoff is that a Protobuf message is opaque bytes you cannot read with your eyes; you need the schema to decode it. Inside the machine room, that trade is usually worth it.

Delivery semantics: how many times did it run?

Return to partial failure. Your stub sent a request and the reply never came. The safe thing to do is retry — but retrying is exactly where "how many times did the call actually execute?" becomes a design decision with three possible answers:

At-most-once — the call runs zero or one times, never more. On a lost reply the client gives up (does not retry). No duplicate execution, but a call can be silently lost.
At-least-once — the client retries until it gets a reply, so the call runs one or more times. Nothing is lost, but the server may execute the same request twice (the first reply was merely lost, not the request).
Exactly-once — the ideal: the effect happens precisely one time. It is not achievable by transport alone; it is manufactured by combining at-least-once delivery with idempotent operations (or deduplication of request IDs).

An operation is idempotent if doing it twice has the same effect as doing it once. "Set the balance to $250" is idempotent — run it five times, the balance is $250. "Add $250 to the balance" is not — run it five times and you have handed the customer $1000 they don't have. This is why real RPC systems attach a unique request ID to each call: the server remembers which IDs it has already processed, and a retried request with a seen ID returns the stored reply instead of running the operation again. That turns a naturally at-least-once transport into an effectively exactly-once effect — the only honest way to get "exactly once."

The single most expensive mistake with RPC is believing the abstraction. A remote call is dressed up to look like getBalance("alice"), so it is desperately tempting to treat it like one: call it in a loop, ignore that it can hang, assume it either fully succeeds or fully fails. It does not. Keep the eight fallacies pinned to the wall — the network is not reliable, latency is not zero, bandwidth is not infinite, the network is not secure. Every remote call needs a timeout (so a dead server can't hang you forever) and usually a retry — and the instant you add retries you have chosen at-least-once delivery, which means duplicates will happen. So any operation you retry must be idempotent (or deduplicated by request ID). "Retry" and "idempotent" are two halves of one thought; add one without the other and you get either lost calls or double-charged customers. RPC is a convenience, not a licence to forget you are on a network.