The Sigmoid Function

Deep inside a classifier is a raw internal score — the result of multiplying features by weights and adding them up. That score could come out as -1000, 253.7, or anything else on the number line, from -\infty to +\infty. But a probability must live between 0 and 1 — you can't be "150% sure" something is spam.

The sigmoid function is the squashing machine that closes that gap. Feed it any real number at all, and it hands back a well-behaved value strictly between 0 and 1 — something you can honestly read as a probability:

\sigma(z) = \frac{1}{1 + e^{-z}}.

Big positive z gives an output near 1; big negative z gives near 0; and z = 0 sits exactly at 0.5 — the "coin flip" midpoint, where the model is completely undecided. Its graceful S-shape turns a raw, unbounded score into a believable probability.

The S-curve

Here is the sigmoid. The slider sharpens or softens the curve: a large steepness makes it snap almost like a hard on/off switch, while a small one leaves it gently sloping. Whatever the steepness, it always stays trapped between 0 and 1 and passes through 0.5 at the centre — that never changes, no matter how the curve is stretched.

Working the formula by hand

The formula looks intimidating, but it's just three steps: negate z, raise e to that power, add 1, then take the reciprocal. Try it at z = 0:

\sigma(0) = \frac{1}{1 + e^{-0}} = \frac{1}{1 + 1} = \frac{1}{2} = 0.5.

Now try a middling positive score, z = 2. Since e^{-2} \approx 0.135, we get \sigma(2) = \frac{1}{1 + 0.135} \approx \frac{1}{1.135} \approx 0.88 — the model is about 88% confident in the positive class. Flip the sign to z = -2 and the roles reverse: e^{2} \approx 7.39, so \sigma(-2) = \frac{1}{1 + 7.39} \approx 0.12 — only 12% confident, i.e. 88% confident in the other class.

Push z even further out and the curve saturates: at z = 10, \sigma(z) \approx 0.99995 — as good as 1. At z = -10, \sigma(z) \approx 0.0000454 — as good as 0. So if a classifier reports \sigma(z) = 0.87, you can read it directly: "87% confident this belongs to the positive class" — spam, malignant, digit-is-a-7, whatever the positive label happens to mean for that model.

A few more values, to build intuition

It helps to see a short table of inputs and outputs side by side, so the shape of the squashing becomes second nature:

Two things jump out. First, the function is symmetric around the midpoint: \sigma(-z) = 1 - \sigma(z), so flipping the sign of the score exactly flips which class you're confident about, while keeping the same strength of confidence — check it above, 0.27 and 0.73 add up to 1, as do 0.12 and 0.88. Second, most of the interesting movement happens in the middle, roughly between z = -3 and z = 3 — outside that band the curve has mostly already made up its mind.

Why this exact shape

The sigmoid is differentiable everywhere — essential, because gradient descent needs a slope to follow. It also has a famously tidy derivative, \sigma'(z) = \sigma(z)\,(1 - \sigma(z)), which keeps the maths clean. Feed a linear score through it and you get logistic regression; stack it inside a network and it becomes an activation function.

There's also a neat way to read the raw score z itself, before it even reaches the sigmoid: it turns out to be the log-odds of the positive class — the logarithm of "how many times more likely positive is than negative." A score of z = 0 means the odds are exactly even (1-to-1, i.e. \sigma = 0.5); every extra unit of z multiplies those odds by a constant factor. The sigmoid is simply the machine that converts those log-odds back into an ordinary 0-to-1 probability.

Seeing \sigma(z) = 0.87 feels like an answer, but on its own it's still just a number between 0 and 1. Turning it into a final "spam" or "not spam" needs one more ingredient: a threshold. The usual choice is 0.5 — anything above counts as the positive class, anything below counts as the negative one — but 0.5 is a choice, not something baked into the sigmoid itself.

A hospital screening test might deliberately pick a threshold of 0.2 instead, so it flags more borderline cases for a doctor to double-check — missing a real illness is far more costly than a false alarm. Exactly where to draw that line is the whole subject of the decision boundary.

Look again at the saturated ends of the S-curve, way out where z is very large or very negative. The curve is nearly flat there — pushing z from 10 to 11 barely nudges \sigma(z) at all, since it's already essentially 1.

That flatness is a problem for learning. Gradient descent nudges weights using the slope of the curve, and a nearly-flat slope means a nearly-zero nudge — the model can get "stuck," barely updating, even though it might still be making mistakes. Stack many sigmoids on top of each other in a deep network and this effect compounds badly, a headache known as the vanishing gradient problem.

This shape wasn't invented for machine learning — it's over 150 years older than that. The Belgian mathematician Pierre François Verhulst used it in the 1830s to model population growth: a population grows fast while resources are plentiful, then the growth slows and levels off as it approaches the environment's limit. Slow, then fast, then slow again — an S.

The very same curve describes how a rumour or a disease spreads through a population (a handful of cases, then a rapid surge, then a plateau once almost everyone who can catch it already has), and how new technology gets adopted — a few early enthusiasts, a fast-growing middle majority, then a slow trickle of latecomers. Wherever something grows quickly but eventually bumps into a ceiling, an S-curve is usually hiding underneath. Machine learning just borrowed a very well-travelled shape.

There's an extra pleasing coincidence buried in the name. In population biology this S-shape is called the logistic curve — which is exactly why feeding it a linear score is called logistic regression. The two "logistics" — population dynamics and machine learning — are completely unrelated fields that happen to share one very useful curve.

"Sigmoid" simply means "shaped like the Greek letter \sigma (sigma)" — or, more loosely today, "S-shaped." It's a whole family name, not one single curve: the hyperbolic tangent \tanh(z) is another sigmoid-shaped function, stretched to run from -1 to 1 instead of 0 to 1, and it shows up constantly in neural networks too. Whenever you hear "sigmoid" in a machine-learning context without qualification, though, people almost always mean this exact 1/(1+e^{-z}) curve — the logistic function.