Imagine hiking down a mountain in thick fog. You can't see the valley floor — you can't even see more than a step or two ahead. But you can feel one thing perfectly well: which direction, right under your boots, points downhill, and roughly how steep it is. So you take a step that way. Then you feel again, and take another step. Repeat that a hundred times, and — fog or no fog — you end up at the bottom of the valley.

Gradient descent is exactly this strategy, applied to a cost function instead of a mountain. The cost surface is a bowl-shaped landscape — you met its shape in visualizing the cost — and the parameters of the model are your position on it. You never see the whole bowl at once. All you ever compute is the slope right where you're standing — the cost's derivative, its gradient — and step downhill from there. Do that again and again, and you walk yourself all the way down to the minimum-cost parameters, one blind step at a time.

Read it like this: the gradient \frac{\partial J}{\partial w} points uphill, in the direction the cost increases fastest — so the minus sign turns you around to face downhill. The number \alpha is the learning rate, how big a step to take. Crucially, the slope tells you two things at once: which way is downhill, and — because a steep slope is a big number and a gentle slope is a small one — roughly how steep it is right here. That single number is why the steps shrink automatically as you approach the bottom: far from the minimum the ground is steep, so the slope is large and the steps are big; near the minimum the ground flattens out, the slope shrinks towards zero, and the steps shrink right along with it. Nobody has to tell gradient descent to slow down — the maths does it for free.

Notice, too, what gradient descent doesn't need. It never asks for a formula for the whole bowl, never peeks ahead to see where the minimum is, and never remembers more than "where am I, and which way is down from here." That's exactly why it works on cost surfaces with millions of parameters, where nobody could ever draw the whole bowl even if they wanted to — the recipe of "measure the local slope, take a proportionally sized step, repeat" scales up completely unchanged whether you have one parameter or one billion.

Worked example: tracing five steps by hand

Let's do the fog-and-boots routine ourselves, with numbers, on the simplest possible bowl: J(w) = (w - 3)^2. Its minimum is obviously at w = 3 (that's where the squared term is zero) — but pretend we don't know that, and let gradient descent find it. The slope is \frac{dJ}{dw} = 2(w - 3). We'll start at w = 0 with a learning rate \alpha = 0.1, and apply w \leftarrow w - 0.1 \times \text{slope} five times in a row.

Look at the last column: 0 \to 0.6 \to 1.08 \to 1.464 \to 1.7712 \to 2.017 — steadily creeping towards the true minimum at 3. And look at the size of each jump: 0.6, 0.48, 0.384, 0.3072, 0.24576 — each one is exactly 0.8 times the last. That's the automatic slowdown from the card above, showing up in real numbers: the further from the minimum, the bigger the slope, the bigger the step; the closer it gets, the gentler everything becomes.

Roll to the bottom

Step the algorithm forward. The dot starts high on the wall, reads the slope under its feet, and takes a step downhill — bigger steps where the bowl is steep, smaller as it nears the flat bottom. Watch it settle into the minimum, where the slope is zero and learning naturally stops. This time the bowl comes from real data points rather than a bare formula — the same idea as the cost function you've already met, just drawn so you can watch the descent happen live instead of computing every round by hand as we did above.

The descent, seen on the full bowl

In two parameters the cost is a 3-D bowl, and gradient descent walks down its inside wall. This rotatable surface shows exactly that — drag it to view the bowl from any angle. The marked trail is a descent path: each step follows the steepest way down, big strides high on the steep wall and ever-smaller ones as the floor flattens, until it settles at the bottom where the slope is zero.

A second climber, a different start — same valley floor

Here's the payoff of the bowl shape. Take the exact same cost, J(w) = (w-3)^2, the same learning rate \alpha = 0.1 — but drop our hiker in at a completely different spot, w = 8, on the other side of the valley.

This time the slope starts out positive (we're uphill on the right side of the bowl now), so every step subtracts a positive number and marches w back down towards 3: 8 \to 7 \to 6.2 \to 5.56 \to 5.048 \to 4.638. Different starting point, opposite side of the bowl, opposite-signed slope at every step — and yet it's heading for exactly the same minimum. That's the guarantee a bowl-shaped (convex) cost gives you: wherever you start, gradient descent walks you to the bottom, not a bottom.

It's worth pausing on why this works so reliably. A bowl shape — formally, a convex function — has exactly one flat spot, and every slope on it points the same way relative to that spot: negative to its left, positive to its right, nowhere else to hide. There is no ridge, no second dip, no decoy valley for the hiker to wander into by mistake. That single structural fact is what turns "follow the slope downhill" from a plausible-sounding heuristic into a genuine guarantee. You could start at w = 100 or w = -100 in our toy example and the story would be identical: a string of steps, each smaller than the last, converging on w = 3.

The engine of all of machine learning

This one idea — follow the slope downhill — trains almost everything: regression, logistic regression, and every neural network ever built. In higher dimensions the gradient is just the slope in every parameter direction at once — imagine standing on a hillside with thousands of compass directions instead of just two, and feeling which combination of all of them points most steeply downhill — and the step adjusts every parameter together. It is, without much exaggeration, the algorithm behind the modern AI boom.

Our two worked examples used one parameter, w, and five rounds you could check on a calculator. A modern large language model repeats the very same three-line loop — read the slope, take a proportional step, repeat — for billions of parameters at once, for millions of rounds. Nobody traces that by hand; the computer does exactly the arithmetic from our tables, just at a scale no human ever could.

See it explained