Imagine stacking layer after layer of neurons, but each one does nothing more than a weighted sum — multiply the inputs by weights, add a bias, done. Ten layers of that, a hundred layers of that... and the network is, mathematically, exactly as powerful as a single layer. All that depth buys you nothing at all. That surprising collapse is the whole reason this page exists.

The fix is to slip a small non-linear "twist" in between every layer's weighted sum and the next layer's input. That twist is called an activation function, and it is the single ingredient that lets stacking layers actually matter — it's what turns a tower of straight lines into something that can bend, curve, and learn genuinely complicated patterns.

This is why activation functions sit at the heart of everything from a spam filter deciding "spam or not spam" to a photo app drawing a box around a face. None of those decisions are simple straight-line splits in the raw data — a spam email doesn't become "more spam" in neat proportion to one number going up. It takes bent, curved decision boundaries to separate the real patterns, and bent boundaries are exactly what a stack of activated layers can build, and a stack of un-activated ones cannot.

Why nonlinearity is not optional

Here's the proof, in miniature. Suppose a tiny two-layer network has no activation function anywhere — just two plain weighted sums, one feeding the next. Say the first layer computes a = 2x + 1, and the second takes that and computes y = 3a - 2. Substitute one into the other:

Look at that result: y = 6x + 1 is itself just a single weighted sum — one weight (6), one bias (1). The second layer added no new power whatsoever; it only relabelled the first layer's weight and bias. This is completely general — chain any number of pure linear layers, \vec{y} = W_2(W_1\vec{x} + \vec{b}_1) + \vec{b}_2 = (W_2W_1)\vec{x} + (W_2\vec{b}_1 + \vec{b}_2), and it always collapses back down to one matrix and one bias vector. A thousand such layers is secretly one layer wearing a very tall costume.

Slip a non-linear activation \sigma in between the layers, though, and the substitution trick breaks: y = 3\,\sigma(2x+1) - 2 cannot be simplified into a single weighted sum of x. The curve genuinely bends. That broken shortcut is precisely what lets a deep network represent things a single layer never could — wiggly boundaries, curved regions, anything more interesting than a straight line or flat plane.

This is not just a maths curiosity. Every real network that recognises handwriting, translates a sentence, or spots a tumour in a scan is built from many stacked layers of neurons. If somebody forgot to put an activation function between those layers, the whole elaborate structure — however many millions of neurons it contains — would be quietly doing the work of a single, embarrassingly simple weighted sum. The activation function is what makes all that depth earn its keep.

The menu of activations

In practice, only a handful of "twists" get used over and over. Each takes one raw number z — the weighted sum coming out of a neuron — and reshapes it:

• Sigmoid, built from the very same sigmoid function used to turn scores into probabilities — squashes any real number into (0, 1). Its S-shaped curve rises smoothly from near 0 for very negative z to near 1 for very positive z, sitting at exactly 0.5 when z = 0.
• Tanh — the same S-shape as sigmoid, but stretched and re-centred to run from -1 to 1 instead of 0 to 1. Being centred on zero (\tanh(0) = 0) often makes it a friendlier choice than sigmoid for hidden layers.
• ReLU ("rectified linear unit") — by far the simplest: \mathrm{ReLU}(z) = \max(0, z). Negative input, output exactly 0; positive input, pass it straight through unchanged. No curve, no exponentials — just a single kink at the origin. Despite (or because of) that simplicity, it is the default first choice in most modern networks.

Switch between all three below and watch how differently they treat the same input — sigmoid and tanh both curl gently towards flat "ceilings" and "floors," while ReLU shoots off in a dead-straight line the moment z turns positive.

Worked example: the same inputs, three different answers

Nothing beats plugging in real numbers. Take three inputs — a fairly negative one, zero, and a fairly positive one — and run each through all three activations:

A few things jump out. At z = -2, sigmoid still reports a small positive number (0.119) — it never truly reaches zero — while ReLU snaps to exactly 0. At z = 3, sigmoid and tanh are already almost pinned to their ceilings (0.953 and 0.995, both very close to their maximum of 1), while ReLU just reports 3 straight back — it never has a ceiling at all. That "no ceiling" property turns out to matter enormously, as the next section explains.

Why ReLU took over

Sigmoid and tanh both flatten out for large |z|, so their slope nearly vanishes there — and a vanishing slope means backpropagation has almost no gradient signal to learn from, stalling deep networks. ReLU's slope stays a constant 1 for every positive input — no matter how large — so gradients keep flowing and training stays fast. It (and cousins like Leaky ReLU and GELU) now powers most modern networks.

The three inputs from the table above make this concrete. Look at the slope (how fast the output changes as z nudges up) at each point, not just the output itself:

By z = 3, sigmoid's slope has shrivelled to 0.045 and tanh's to a nearly-flat 0.010 — each step of learning there moves the weights by only a tiny fraction of what it would near the centre. ReLU's slope, by contrast, is still a full 1, exactly as strong as it was at z=0.01. Stack many saturated layers together and those tiny slopes multiply together into something vanishingly small — the heart of why deep sigmoid/tanh networks were once notoriously hard to train, and why swapping in ReLU was such a turning point.

Choosing an activation in practice

With three options on the table, which one goes where? A common recipe: use ReLU in every hidden layer, purely because it trains fast and rarely saturates. Then, at the very last layer — the one that actually produces the network's final answer — pick an activation that matches the question being asked, not the training speed. A yes/no probability (is this email spam?) wants sigmoid, since its output lands neatly in (0, 1) and can be read directly as a chance. A choice between many categories (which of ten digits is this?) wants a cousin of sigmoid called softmax, which you'll meet later. Tanh sees less use today, but still shows up in a few specialised corners, such as inside recurrent networks that process sequences.

Notice the pattern: it's rarely "one activation for the whole network." Hidden layers optimise for trainability; the output layer optimises for matching the shape of the answer.