The Gradient

Imagine standing on a foggy hillside with an altimeter and no map. You cannot see the summit — you cannot see three metres — yet at the spot where you stand there is one direction in which the ground climbs fastest, one number saying how steep that climb is, and one curve of "equal height" threading through your boots. Remarkably, a single vector encodes all three at once. It is the compass needle of multivariable calculus: everywhere you plant it, it swings to point straight uphill.

The directional derivative gave us this vector without naming it. Collecting the two partials of f(x, y) into a single vector defines the gradient:

\nabla f = \left( f_x,\; f_y \right).

It is far more than bookkeeping. The gradient is a genuine arrow living in the xy-plane, and that arrow has three superpowers: it points the way steepest uphill, its length is exactly how steep that climb is, and it always stands perpendicular to the level curves. Everything a point could want to know about how f changes nearby — the whole local story — is packed into this one arrow. All three properties fall out of one identity, which we derive next.

(The symbol \nabla is pronounced "nabla" or "del". The name comes from an ancient Phoenician harp of that triangular shape — a physicist friend of James Clerk Maxwell suggested it as a joke, and it stuck.)

Deriving the three properties

Everything starts from the directional-derivative formula and a single fact about dot products. No new machinery is needed — just careful looking.

Step 1 — write the rate as a dot product. For a unit direction \mathbf{u},

D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}.

Step 2 — turn the dot product into a cosine. For any two vectors, \mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta. Here \|\mathbf{u}\| = 1, so with \theta the angle between \nabla f and \mathbf{u},

D_{\mathbf{u}} f = \|\nabla f\|\,\|\mathbf{u}\|\cos\theta = \|\nabla f\|\cos\theta.

Pause on this. As you stand at a point and slowly turn on the spot, the slope you feel underfoot is \|\nabla f\|\cos\theta — a fixed number times a cosine. The whole compass of possible slopes at one point is a single cosine wave.

Step 3 — maximise over directions. As \mathbf{u} swings around, the only thing that changes is \cos\theta, which is largest when \theta = 0 — that is, when \mathbf{u} points along \nabla f. There \cos\theta = 1 and

\max_{\mathbf{u}} D_{\mathbf{u}} f = \|\nabla f\|.

So \nabla f points in the direction of steepest ascent, and the steepest slope is its magnitude \|\nabla f\|. Two bonus readings come free: at \theta = 180^\circ, \cos\theta = -1, so the opposite direction -\nabla f is steepest descent with rate -\|\nabla f\|; and at \theta = 90^\circ, \cos\theta = 0 — stepping sideways to the gradient, the ground is momentarily flat. That last reading is the seed of the third property.

Step 4 — set up the level curve. A level curve is the set where f stays constant — one contour line on the map. Walk along it with a path \big(x(t), y(t)\big), so that

f\big(x(t), y(t)\big) = c \quad\text{(constant) for all } t.

Step 5 — differentiate the constant. Along the contour, nothing changes — that is the whole point of a contour. The right side is constant, so its derivative is zero; the left side opens up by the chain rule:

\frac{d}{dt}\, f\big(x(t), y(t)\big) = f_x\, x'(t) + f_y\, y'(t) = 0.

Step 6 — recognise the dot product. That sum is the gradient dotted with the path's velocity \mathbf{T} = (x', y'), which is the tangent to the level curve:

\nabla f \cdot \mathbf{T} = 0.

A zero dot product means perpendicular. The gradient is at right angles to the level curve through every point — it always points straight "across the contours", never along them. On a topographic map you can now see the gradient field without computing anything: at every point, rotate the contour line a quarter turn. And where contour lines crowd together, the hill is steep — so \|\nabla f\| is large exactly where the contours are dense.

Let f be differentiable at a point with \nabla f \neq \mathbf{0} there. Then:

Steepest ascent. \nabla f points in the direction in which f increases fastest.
Maximum rate. That fastest rate of increase equals the gradient's magnitude, \|\nabla f\| (and the steepest descent is -\nabla f, with rate -\|\nabla f\|).
Perpendicular to level sets. \nabla f is orthogonal to the level curve f = c through the point — because f does not change along that curve, so \nabla f \cdot \mathbf{T} = 0 for the tangent \mathbf{T}.

Worked example 1 — the bowl

Let f(x, y) = x^2 + y^2 at the point (3, 4).

Step 1 — the gradient. f_x = 2x, f_y = 2y, so \nabla f(3, 4) = (6, 8).

Step 2 — direction of steepest ascent. Straight along (6, 8) — radially outward from the origin, which makes sense: f is a bowl and the fastest way up is directly away from the bottom.

Step 3 — maximum rate. \|\nabla f\| = \sqrt{6^2 + 8^2} = \sqrt{100} = 10.

Step 4 — perpendicularity check. The level curves of x^2 + y^2 are circles centred at the origin; their tangents are perpendicular to the radius, and (6, 8) is exactly the radial direction. The gradient crosses the contour at a right angle, as promised.

Worked example 2 — a hill, checked to the last dot product

Now a hill that is not round, so nothing is obvious by symmetry:

h(x, y) = 100 - x^2 - 2y^2,

the height in metres above the point (x, y), with a 100-metre summit at the origin. You stand at (3, 2), where the height is h(3,2) = 100 - 9 - 8 = 83 m.

The gradient. h_x = -2x and h_y = -4y, so

\nabla h(3, 2) = (-6, -8).

Direction of steepest ascent. Both components are negative: the fastest way up is towards decreasing x and decreasing y — roughly towards the summit at the origin, but not exactly at it. As a unit vector, the direction is \tfrac{1}{10}(-6, -8) = (-0.6, -0.8), whereas the straight line to the summit points along (-3, -2) normalised, about (-0.83, -0.55). On a lopsided hill the steepest step and the beeline to the top genuinely disagree — the hill falls away faster in y (that factor of 2), so the gradient leans harder into y.

Maximum rate. \|\nabla h\| = \sqrt{(-6)^2 + (-8)^2} = \sqrt{100} = 10: the ground climbs 10 metres per metre stepped — a cliff-like 84^\circ pitch. Step the other way, along -\nabla h = (6, 8), and you descend at exactly that rate.

Perpendicularity, verified explicitly. The level curve through your feet is 100 - x^2 - 2y^2 = 83, i.e. the ellipse x^2 + 2y^2 = 17. Differentiate implicitly to find its tangent slope:

2x + 4y\,\frac{dy}{dx} = 0 \quad\Longrightarrow\quad \frac{dy}{dx} = -\frac{x}{2y} = -\frac{3}{4} \text{ at } (3,2).

A slope of -\tfrac{3}{4} means the tangent vector is \mathbf{T} = (4, -3). Now dot it with the gradient:

\nabla h \cdot \mathbf{T} = (-6)(4) + (-8)(-3) = -24 + 24 = 0.

Zero, on the nose. The gradient really does cross the contour ellipse at a perfect right angle — no symmetry argument needed, just arithmetic.

Worked example 3 — the heat-seeking bug

The gradient is not only for hills — it works for any scalar field. Suppose the temperature on a metal plate is

T(x, y) = 70 - x^2 + 2xy \quad (\text{degrees}),

and a cold little bug sits at (1, 2). Which way does it crawl to warm up fastest?

Compute the gradient. T_x = -2x + 2y and T_y = 2x, so at the bug's spot

\nabla T(1, 2) = (-2 + 4,\; 2) = (2, 2).

Read it off. The bug crawls along (2, 2) — north-east, at exactly 45^\circ — and the temperature rises at \|\nabla T\| = \sqrt{4 + 4} = 2\sqrt{2} \approx 2.8 degrees per unit crawled, the best possible rate in any direction. An overheating bug at the same spot crawls along -\nabla T = (-2, -2): steepest descent, cooling at 2\sqrt 2 degrees per unit. And a bug that is perfectly comfortable crawls along the isotherm — perpendicular to \nabla T, along (1, -1) or (-1, 1) — feeling no temperature change at all.

One arrow, three life strategies: chase it, flee it, or circle it. That is the gradient's whole résumé.

The most-drawn wrong picture in this subject is a gradient arrow tilting up the 3-D surface like a ski pole. It isn't there.

\nabla f lives in the input plane. For f(x, y) it is a flat 2-D vector, (f_x, f_y), drawn on the floor of the map — on the contour plot, not on the mountain. It says "walk this way across the map and f climbs fastest"; the climbing happens in the value of f, not in the arrow. (For f(x,y,z) the gradient is a 3-D vector in the input space, perpendicular to level surfaces — same story, one dimension up.)
\nabla f = \mathbf{0} flags a flat spot. At a summit, a valley bottom, or a mountain-pass saddle there is no uphill direction to point in, so the gradient has nothing to say and collapses to the zero vector. Those points get a whole theory of their own on the next page.
Sign discipline. \nabla f is steepest ascent; steepest descent is -\nabla f. Writing the wrong sign sends your heat-seeking bug to the freezer.

Watch it stay perpendicular

The rings are level curves of f(x, y) = x^2 + y^2 (circles). Slide the point around its ring: the gradient arrow \nabla f = (2x, 2y) always points straight outward, crossing the contour at a perfect right angle no matter where P sits.

Now play with the second slider. Moving P to an outer ring makes the arrow grow: its length \|\nabla f\| = 2\sqrt{x^2 + y^2} = 2r is the steepest slope there, and the bowl gets steeper the farther you climb from the bottom. Notice the dashed tangent line, too — that is the direction a "comfortable bug" would walk, the one direction at P along which f does not change at all. Gradient across, tangent along: the two directions between them tell you everything about the local landscape.

Steepest ascent, on the hill itself

Contours flatten the story onto the floor; here is the hill it came from. This is a smooth bump z = f(x, y) with its summit over the origin — drag to rotate it. The marked point sits on the slope, and the short segment leaving it points in the direction of steepest ascent: straight uphill, toward the peak. That uphill heading, read on the floor below, is exactly \nabla f.

If \nabla f is the steepest way up, then -\nabla f is the steepest way down — and rolling downhill is how almost every machine-learning model is trained. Gradient descent repeatedly nudges a point against its gradient,

\mathbf{x}_{n+1} = \mathbf{x}_n - \eta\, \nabla f(\mathbf{x}_n),

where the small step size \eta is the "learning rate". Each step lowers f (for small enough \eta), and the process coasts to a halt exactly where \nabla f = \mathbf{0} — a critical point, the subject of the next page.

Now scale it up. Training a neural network means minimising an error function of not two variables but billions — one per adjustable weight — and the recipe is still this one line, run trillions of times across the world's data centres every day. Nobody can picture a billion-dimensional error landscape, and nobody needs to: the gradient is computed, the step is taken downhill, and the network improves. It is a fair claim that \mathbf{x} - \eta \nabla f is the single most-executed piece of vector calculus on the planet. The gradient is what makes the impossible-to-visualise navigable.

Long before anyone wrote \nabla, the universe was computing it:

Rivers carve paths of steepest descent — water at each point accelerates along -\nabla h of the terrain height, which is why streams cross contour lines on a map at right angles.
Heat flows from hot to cold along -\nabla T — Fourier's law says the heat flux is \mathbf{q} = -k \nabla T, temperature's own gradient descent.
Diffusion obeys Fick's law, -D \nabla c: perfume spreads across a room by sliding down its own concentration hill.
Bacteria really do play the heat-seeking-bug game: in chemotaxis, E. coli swims up nutrient gradients (and down toxin gradients) by sampling concentrations as it tumbles along — a living gradient estimator about two micrometres long.