The previous lesson ended with a GPU drinking terabytes per second from its own HBM. Here is the uncomfortable sequel: one GPU is never enough. A frontier language model's weights alone overflow any single device's memory; its training run wants thousands of GPUs computing as one machine for months. And the moment two GPUs must act as one, a new organ appears on the die and in the system — the interconnect — and it is designed with exactly the same seriousness as the memory system, because it fails the same way: silently, by making world-class arithmetic units wait.

This lesson climbs the ladder: the PCIe baseline and its ceiling, direct GPU-to-GPU links and the switched fabrics built from them, and then the traffic pattern that shaped all of it — the all-reduce — whose arithmetic is so tidy you can carry it in your pocket.

The baseline: PCIe, and its ceiling

Every GPU already has a link to the world: PCIe, the general-purpose peripheral bus. A generous x16 Gen5 slot moves about 64 GB/s each way — perfectly respectable for loading textures or streaming datasets. Now put it next to the numbers it must serve: the GPU's own HBM runs at several thousand GB/s. Traffic between two GPUs over PCIe typically also detours through the CPU's root complex, sharing lanes with storage and NICs. The mismatch is roughly two orders of magnitude: any algorithm that shuffles serious data between GPUs over PCIe turns a computing machine into a waiting machine.

So GPU vendors did to the interconnect what they had already done to memory: built a dedicated one. NVLink-class direct links (AMD's Infinity Fabric plays the same role) connect GPUs point-to-point at hundreds of GB/s per GPU — modern parts aggregate close to a TB/s of link bandwidth, within an order of magnitude of local HBM. And the links are memory-semantic: a GPU can issue plain loads, stores and atomics into a peer's HBM, as if it were a slightly-farther region of its own address space — no packet assembly, no driver round-trip, no bounce buffer in host RAM. Peer memory becomes just another rung on the memory hierarchy: registers, L1, L2, local HBM, neighbour's HBM.

Point-to-point links alone, though, give a fixed budget an awkward shape: with 8 GPUs and a handful of links each, you cannot wire every pair directly at full rate. Enter the switch: route every GPU's links into a crossbar chip (an NVSwitch, in NVIDIA's fabric), and every GPU can talk to every other GPU at its full link rate, all at once — an all-to-all fabric inside the chassis. This is the interconnection-network story you already know — topology, bisection bandwidth, routing — reincarnated with GPUs at the endpoints and gradient tensors as the traffic.

The traffic that shaped the fabric: all-reduce

Why does GPU-to-GPU traffic deserve custom silicon? Because deep-learning training generates one pattern above all. In data-parallel training (see ) every GPU holds a full copy of the model and computes gradients on its own slice of the batch; before anyone can take an optimiser step, all GPUs must average their gradients — every GPU needs the sum of every GPU's array. That collective operation is the all-reduce, and during training it fires after every batch, moving the entire gradient — gigabytes — each time. (Tensor parallelism — see — adds collectives inside every layer, which is even less forgiving.)

The naive scheme is a disaster: every GPU sends its full D-byte array to every other — D(N-1) bytes out of each GPU. The classic fix is the ring all-reduce. Arrange the N GPUs in a logical ring and chop each array into N chunks of D/N bytes:

• Reduce-scatter — for N-1 steps, every GPU sends one chunk to its right neighbour and adds the chunk arriving from its left into its own copy. After N-1 steps, each GPU holds one chunk of the fully-summed result.
• All-gather — the finished chunks circulate around the same ring for another N-1 steps — pure copying, no more adding — until everyone has all of them.

Count the traffic: 2(N-1) steps, one D/N-byte chunk sent per GPU per step:

Feel it with numbers: 8 GPUs averaging an 8 GB gradient. Naive: each GPU sends 8 \times 7 = 56 GB. Ring: 2 \cdot 8 \cdot 7/8 = 14 GB — and at 400 GB/s of link bandwidth that's ~35 ms per batch, hidden further by overlapping communication with the backward pass.

Watch the ring turn

Four GPUs, each starting with its own 4-chunk gradient. Step through the two phases and watch where the finished chunks appear.

Beyond the ring: trees, switches that do maths, and NICs that skip the CPU

The ring's weakness is its 2(N-1) serial steps: for a small tensor on a thousand-GPU ring, latency — not bandwidth — dominates. So real systems keep a menu. Tree all-reduce runs in O(\log N) steps — reduce up a tree, broadcast back down — winning on latency at some cost in link balance. Hierarchical schemes exploit the topology's tiers: a fast ring inside each chassis over the switched fabric, then a tree across chassis over the cluster network, so the slow wires carry only one chassis-worth of pre-reduced data.

Two further tricks push work off the GPUs entirely. In-network reduction teaches the switch itself to add: gradient packets from many GPUs meet inside the switch, which sums them on the fly and forwards one result — the data crosses the network once instead of circulating. And across nodes, RDMA with GPUDirect lets the network card DMA straight out of one GPU's HBM into a remote GPU's HBM — no copy into host memory, no CPU on the data path at all.

Stitching this together is the job of the collective-communication library (NCCL and its cousins): at startup it discovers the topology — which GPUs share a switch, which NICs sit near which GPUs, what the tiers look like — and chooses the algorithm per call: ring here, tree there, hierarchical across the cluster, in-network where the switches offer it. The same training script hits wildly different wire schedules on different machines, and never knows.

Simulate the ring

Twenty lines of TypeScript replace a wall of diagrams. The simulator walks both phases, counts what each GPU sends, and checks the count against the formula. Try N = 128 and watch the per-GPU total creep towards — but never reach — 2D.