For linear regression the cost was a perfect bowl — smooth, single-bottomed, and however you start rolling downhill you always end up in the same place: the global minimum. That shape is not an accident; it comes from a cost built out of squared errors of a linear model, which is mathematically guaranteed to be bowl-shaped (convex) no matter what the data looks like.

A neural network throws that guarantee away. Stack several layers, glue them together with non-linear activations, and multiply it all out across millions of adjustable weights, and the cost — now usually called the loss — stops being a bowl. Plotted against its weights it becomes a wild, high-dimensional landscape: hills, ridges, deep canyons, wide flat plains, and countless valleys of different depths. A surface like this is called non-convex.

Gradient descent still works on this terrain — it is, at heart, a foggy hiker feeling for the steepest way down and taking a step. The hiker will still get somewhere lower than they started. What's gone is the guarantee that "somewhere lower" means the single lowest point on the entire map. On a landscape this complicated, where you end up depends heavily on where you began.

"High-dimensional" is not an exaggeration for effect — it's the literal shape of the problem. Even a small network with two hidden layers of a hundred neurons each has tens of thousands of weights, which means its loss is a function of tens of thousands of variables, plotted over a landscape with tens of thousands of dimensions. A modern industrial-scale network can have billions. Our foggy hiker isn't walking over rolling hills; they're feeling their way down a mountain that exists in a space no picture could ever fully show.

Meet the terrain: three classic traps

Every foggy hiker on a bumpy landscape eventually meets the same three obstacles. They all share one symptom — the ground underfoot goes flat, so the gradient (the slope the hiker can feel) shrinks towards zero, \nabla L(w) \approx 0 — but they are geometrically very different, and telling them apart matters for understanding why training sometimes stalls.

• Local minimum. A valley whose floor is lower than everywhere immediately around it — so gradient descent has genuinely nowhere better to step — but which is not the lowest point anywhere on the whole landscape. Standing in it, every direction feels like "up," so the hiker (wrongly) concludes they've reached the bottom of the world.
• Saddle point. Named after a horse's saddle: curving up if you step along one direction, but curving down if you step along another. In the millions of directions available inside a real network, a saddle point somewhere is almost guaranteed, and because the gradient vanishes there too, gradient descent can crawl along the flat directions for a long time before it finds the one direction that actually goes further down.
• Plateau. Not a dip at all — just a wide, nearly-flat plain. The true minimum might be close by, but with the slope almost zero in every direction, each gradient-descent step is tiny, and progress can crawl to what looks, from the training curve, exactly like a standstill.

Notice the family resemblance: all three are places where the gradient shrinks toward zero, which is exactly the signal gradient descent uses to decide "keep going" versus "stop, you've arrived." A local minimum is a trap gradient descent is right to stop at (locally, there's nowhere better) but wrong about globally. A saddle point or a plateau is worse in a sense — the gradient is small, but not because the algorithm has actually arrived anywhere meaningful; it's just temporarily lost the signal it needs to keep moving.

A bumpy descent

The curve below is a simple 1‑D stand-in for a slice through a real loss landscape: notice it has two dips — a shallower one on the left and a deeper, true minimum further along. Drag the starting position and watch the ball roll downhill into whichever valley happens to be nearest.

Start the ball just to the left of the shallow dip and it rolls a short distance and stops — the slope either side of it points back inward, so as far as gradient descent can tell, that dip is the bottom of the world. Start it further right, past the little bump that separates the two dips, and the exact same rolling rule carries it all the way down into the true, deeper minimum instead. Nothing about the algorithm changed — only the starting point did.

Notice, too, that the ball never "knows" it settled for less. There is no step in gradient descent that compares the shallow dip's floor to the deep dip's floor — the algorithm only ever asks "which way is downhill from exactly where I am right now?" Two starting points a hair's breadth apart, on either side of the little bump between the dips, can end their journeys a long way apart in loss — one merely acceptable, the other excellent. That sensitivity to the starting point is the price gradient descent pays for having no map of the whole landscape, only a compass that reads "downhill."

The terrain, in the round

Training searches a high-dimensional loss landscape for a low valley. Even this cartoon in just two weights has several dips and ridges — drag the surface to explore the terrain, and you can see exactly why a foggy hiker feeling only for "downhill" can settle into the wrong basin.

Why it works anyway

Here is the happy surprise that reassured a lot of worried researchers in the 2010s. In a landscape with only one or two weights, like the cartoon curve above, getting trapped in a shallow local minimum is a real and common disaster. But real networks have millions of weights, so the landscape has millions of dimensions — and something strange happens as the dimension count climbs.

For a point to be a bad local minimum, the surface must curve upward in every single one of those millions of directions at once. As the number of directions grows, that becomes astronomically unlikely — there is almost always at least one direction that still slopes gently downward. Detailed studies of real, trained networks back this up: the vast majority of local minima found in practice sit at a loss value very close to the best one found anywhere, so getting stuck in a badly bad valley turns out to be rare.

What trips up training far more often, it turns out, is not local minima at all — it's saddle points. Because a saddle only needs curvature that goes up in some directions and down in others, they are far more common than genuine minima in high dimensions, and a network can spend a long time wandering across the nearly-flat directions near one before it finds its way out and continues downhill.

There's a neat intuition for why saddles so outnumber true minima as the dimension count grows. Imagine flipping a coin once for every direction in the landscape — heads means "curves up here," tails means "curves down here." A genuine local minimum needs every single coin to land heads at once; a saddle point just needs a mix. With two or three directions, all-heads is a fair bet. With ten thousand directions, all-heads is astronomically unlikely, while "a mix of heads and tails" is essentially certain — so almost every flat-gradient point a large network stumbles into is a saddle, not a minimum.