A network with a million weights just made a prediction — and it was wrong. Somewhere in that tangle of a million numbers, some weights nudged the answer the wrong way more than others. Which ones deserve the blame, and by how much? Check every weight one at a time, holding all the others still, and you'd need roughly a million separate mini-experiments just to train one step. For a network with a billion weights, forget it — the sun would burn out first.

Backpropagation solves this "credit assignment" problem with breathtaking efficiency. It computes the exact gradient — how much every single weight contributed to the error — in one single backward sweep through the network, using nothing more exotic than the chain rule you already know from calculus, applied layer by layer.

The network is a chain of functions, so the chain rule says the gradient at an early layer is the product of the local slopes of every layer after it. Backprop computes the error at the output, then propagates it backward, layer by layer — each layer multiplying the incoming gradient by its own local slope and passing the result further back.

The brute-force alternative, and why it can't scale

Before seeing the clever way, it's worth seeing the naive way, if only to appreciate why backpropagation was such a breakthrough. Suppose you didn't know the chain rule trick. You could still estimate one weight's gradient by numerical differentiation: nudge that one weight by a tiny amount \varepsilon, rerun the entire forward pass to see the new loss, and divide the change in loss by \varepsilon. That gives a decent estimate of how much that one weight matters.

The catch: to get gradients for every weight this way, you'd have to repeat that whole nudge-and-rerun process once per weight, separately.

That is not a small difference — it's the difference between a training step that finishes in a heartbeat and one that would still be running long after everyone involved has retired. Backpropagation gets the exact same information (the gradient for every weight) as the brute-force method, but for the price of roughly two passes through the network instead of a million. That efficiency gap is precisely why deep learning became practical only once backpropagation was in every toolbox.

Why backward? The blame relay

Here is the key trick, and it's why the algorithm runs backward instead of forward. To know how much a late weight (close to the output) is to blame, you only need the error itself — it's right there. But to know how much an early weight (close to the input) is to blame, you first need to know how much blame landed on the layer just after it — because the early weight's only influence on the final error runs through that later layer.

So the layers hand a "blame signal" backward like a relay baton: the output layer computes its own gradient first, then passes a transformed version of the error back to the layer before it, which computes its gradient and passes its own transformed signal back again, and so on — all the way to the input. Step through the two phases below: first the forward pass (blue) that produces a prediction, then the backward pass (orange) that relays the blame.

Worked example: blaming a tiny two-layer network

Let's make "the relay" completely concrete with real numbers. Take the smallest network that still has the essential shape: one input, one hidden unit, one output, connected by two weights.

Say x = 2, and the network's current weights are w_1 = 0.5 and w_2 = 3 (ignore activation functions for a moment — just plain multiplication at each step, to keep the arithmetic in view). The forward pass is:

But the true answer was y = 5. The network is wrong by a lot. Using the squared error L = \tfrac12(\hat y - y)^2, the loss is \tfrac12(3-5)^2 = 2. Now the backward pass relays the blame, one chain-rule step at a time — notice each row needs the row above it:

Look at the order: w_2's gradient is computed first, straight from the output error. Only then can w_1's gradient be computed, because it depends on \partial L/\partial h — a quantity that didn't exist until the layer after it had finished. That's the whole reason for going backward: each layer's blame is built from the layer that comes after it, so the layers must be visited in reverse. Both gradients are negative, which tells gradient descent to increase both weights — sensible, since the network's guess of 3 needs to climb toward the true 5.

Real networks add one more ingredient at each layer: a nonlinear activation function applied right after the multiplication (without one, stacking layers would collapse into nothing more than one big multiplication). The chain rule shrugs this off exactly the same way — the activation's own local slope simply becomes one more factor multiplied into the relayed gradient at that layer, alongside the weight. The bookkeeping grows a little; the underlying idea, multiply the local slopes together as you relay backward, never changes.

The payoff: one sweep, every weight, at once

Scale the two-layer toy up to a real network with a hundred layers and a hundred million weights, and nothing about the idea changes — only the bookkeeping grows. One backward sweep, computed in roughly the same time as the forward pass that made the prediction, produces the exact gradient for every single weight, simultaneously. That single sweep is then handed straight to gradient descent, which uses it to nudge every weight downhill at once.

Without this trick, training any network bigger than a toy would be computationally hopeless. With it, training networks with billions of weights — the kind behind modern image recognition and language models — is merely expensive, not impossible. Backpropagation is what turned neural networks from a cute idea into an engineering discipline.

Nothing about the argument required the network to be a plain, single-file chain, either. Real architectures branch, merge, and reuse the same weight in several places (a convolutional filter slid across an entire image is one famous example). The chain rule handles all of this just as happily — a branch simply means a weight's total blame is the sum of the blame relayed back along every path it influenced. The two-layer toy above and a billion-weight network with dozens of branching paths are solved by exactly the same rule, just applied more times.

See it explained