Forward Propagation

Feed a network an input, and it doesn't produce an answer by magic — it pushes that input straight through the whole structure, one layer at a time, each layer's output becoming the next layer's input, until a prediction pops out the other end. That single pass, start to finish, is called forward propagation — or just the "forward pass."

This is exactly what happens, right now, every single time a trained network is actually used. A spam filter reads an email and forward-propagates it to a spam/not-spam score. A self-driving car's camera feeds a frame in and forward-propagates it to "pedestrian" or "clear road." A voice assistant forward-propagates a slice of audio into the word it just heard. Whatever the task, the mechanism underneath is the same one you'll trace by hand on this page.

The mechanics, layer by layer

Every layer does exactly the same two-step job. First it takes the values arriving from the layer before it (or the raw input, for the very first layer) and combines them into a weighted sum for each of its own neurons — multiply by weights, add a bias:

\vec{z} = W\vec{x} + \vec{b}

Second, it squashes that raw weighted sum through an activation function \sigma before handing the result onward:

\vec{a} = \sigma(\vec{z}) = \sigma(W\vec{x} + \vec{b})

Repeat that "weighted sum, then activation" pair once per layer, feeding each layer's \vec{a} in as the next layer's \vec{x}, and you've walked the whole network:

\vec{a}^{(1)} = \sigma(W_1\vec{x} + \vec{b}_1), \quad \vec{a}^{(2)} = \sigma(W_2\vec{a}^{(1)} + \vec{b}_2), \quad \dots

There's nothing more to it than that — the whole network is one big function, built entirely by chaining these matrix multiplies and squashes together, layer after layer, until the last one produces the final answer.

Worked example: tracing a tiny network by hand

Numbers make this concrete. Take a small network: 2 inputs, feeding a hidden layer of 2 neurons using ReLU, feeding a single output neuron using sigmoid (so the final answer reads like a probability). Say the input is \vec{x} = (1, 2), and the trained weights happen to be:

W_1 = \begin{pmatrix} 2 & 1 \\ -3 & 1 \end{pmatrix}, \quad \vec{b}_1 = \begin{pmatrix} -1 \\ 0 \end{pmatrix}

Step 1 — the hidden layer's weighted sum. Multiply each row of W_1 by \vec{x} and add the matching bias:

z_1 = (2)(1) + (1)(2) + (-1) = 3, \qquad z_2 = (-3)(1) + (1)(2) + 0 = -1

Step 2 — apply ReLU. ReLU passes positives through unchanged and clips negatives to zero:

a_1 = \mathrm{ReLU}(3) = 3, \qquad a_2 = \mathrm{ReLU}(-1) = 0

Notice a_2 got clipped all the way to 0 — that neuron simply switched off for this particular input. Only a_1 = 3 carries information onward.

Step 3 — the output layer's weighted sum. With W_2 = (0.5, -0.5) and b_2 = 0.2:

z_3 = (0.5)(3) + (-0.5)(0) + 0.2 = 1.7

Step 4 — apply sigmoid. Squash that raw score into a probability:

\hat{y} = \sigma(1.7) = \frac{1}{1 + e^{-1.7}} \approx 0.846

Start to finish: the input (1, 2) forward-propagated through two layers to a final prediction of about 0.846 — an 85% chance of whatever "yes" means for this network. Every intermediate value (z_1, z_2, a_1, a_2, z_3) was needed along the way, but only \hat{y} is the answer anyone actually wants. Step through the same computation in the diagram below.

function relu(z: number): number { return Math.max(0, z); } function sigmoid(z: number): number { return 1 / (1 + Math.exp(-z)); } const x = [1, 2]; const W1 = [[2, 1], [-3, 1]]; const b1 = [-1, 0]; const W2 = [0.5, -0.5]; const b2 = 0.2; const z1 = W1.map((row, i) => row[0] * x[0] + row[1] * x[1] + b1[i]); const a1 = z1.map(relu); const z2 = a1[0] * W2[0] + a1[1] * W2[1] + b2; const yHat = sigmoid(z2); console.log("hidden weighted sums:", z1); console.log("hidden activations:", a1); console.log("output prediction:", yHat.toFixed(3));

Why "forward"?

The name is a direction, not a decoration. Information flows strictly one way here — from the input end of the network toward the output end, never backward, and never sideways between neurons in the same layer. That one-way flow is exactly what makes forward propagation cheap: each layer only ever needs the finished output of the layer before it, so a whole network's prediction is just one clean pass from front to back.

That name also sets up a contrast worth remembering. Once the network is done predicting, training it means sending information the opposite way — from the error at the output, backward through the layers, to work out how every single weight should change. That reverse journey has its own name, backpropagation, and it's a genuinely different computation built directly on top of the one you just traced by hand.

Fast, but only half the story

Forward propagation is cheap and parallel — a few matrix multiplies — which is why a trained network can label an image in milliseconds. Run a whole batch of inputs at once and it becomes a single matrix–matrix multiply. But this only uses a network; it doesn't teach it. For that we need to measure the error and send it backwards — first by mapping out the loss landscape, then by backpropagation.

Two things trip people up about the forward pass:

The activation must be applied after EVERY weighted sum — not just tacked on at the very end. Skip the activation on a hidden layer, and that layer collapses into a plain linear transform; stack several such un-activated layers together and the whole network mathematically reduces to a single layer, no matter how many you stacked (the same trap covered in activation functions). Forward propagation only builds something genuinely powerful when every hidden layer gets its non-linear squash.
Forward propagation doesn't involve any learning. It's easy to assume that running data through a network is somehow "training" it — it isn't. A forward pass just uses whatever weights the network already has, whether they're carefully trained or freshly initialized to random junk. Feed the same input through an untrained network and a trained one, and you'll get two very different, equally "computed" answers — only one of them is any good, and forward propagation alone has no way of knowing which.

The exact hand-traced computation above — multiply, add, squash, repeat — is precisely what happens every time a photo app instantly draws a box around a face, or a voice assistant transcribes a sentence the moment you stop talking. A trained network's forward pass can run in a few milliseconds, even on a phone, because it's nothing more exotic than a cascade of matrix multiplications.

In the industry, running forward propagation on new, real-world data (after training is long finished) has its own name: inference. It turns out to be such a huge, separate engineering challenge — running billions of these forward passes a day, as cheaply and quickly as possible — that entire specialised chips exist purely to make inference fast, distinct from the chips used to train the network in the first place. Training happens once (or occasionally); a popular model's forward pass might run trillions of times over its lifetime, so shaving even a fraction of a millisecond off matters enormously at that scale.