Multivariable Optimization

Picture flying a drone over a mountain range. Some spots are peaks — the ground falls away no matter which way you step. Some are pits — valley floors where every step leads up. But there's a third, sneaky kind of flat spot: a mountain pass, level under your feet yet neither a summit nor a bottom — it rises toward the peaks flanking it on one axis and dips toward the valleys on the other. Calculus calls that a saddle point, and hunting for maxima and minima in two variables means learning to tell all three apart.

In one variable, hills and valleys sit where the derivative vanishes. The same instinct works in two variables — but with a twist. At a peak or a pit of a surface, the ground is momentarily flat in every direction, so every directional derivative is zero. By the gradient identity D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}, that forces the whole gradient to vanish:

\nabla f = (f_x,\, f_y) = (0,\, 0).

A point where \nabla f = \mathbf{0} is a critical point — the multivariable cousin of a critical point in single-variable calculus. The twist is the new third option: besides a local max or min, the surface can do something a curve never can — rise one way and fall the other, forming a saddle.

The second-derivative test

Finding \nabla f = \mathbf{0} locates the candidates; classifying them needs the second derivatives. They assemble into the Hessian, the matrix of second partials, and the test hinges on its determinant, the discriminant:

D = \det \begin{pmatrix} f_{xx} & f_{xy} \\ f_{xy} & f_{yy} \end{pmatrix} = f_{xx}\, f_{yy} - f_{xy}^{\,2}.

At a critical point, the sign of D says whether the surface curves the same way in every direction (a genuine extremum) or in opposite ways along two directions (a saddle); then the sign of f_{xx} tells max from min.

Let (a, b) be a critical point of a twice-differentiable f (so \nabla f(a, b) = \mathbf{0}), and write D = f_{xx} f_{yy} - f_{xy}^{\,2} evaluated at (a, b). Then:

D > 0 and f_{xx} > 0 ⟹ local minimum (a bowl: concave up every way).
D > 0 and f_{xx} < 0 ⟹ local maximum (a dome: concave down every way).
D < 0 ⟹ saddle point (up one way, down another — no extremum).
D = 0 ⟹ test inconclusive (the second order isn't decisive; look closer).

A worked classification

Classify every critical point of f(x, y) = x^3 - 3x + y^2.

Step 1 — the gradient. Differentiate in each variable:

f_x = 3x^2 - 3, \qquad f_y = 2y.

Step 2 — solve \nabla f = \mathbf{0}. Setting both to zero: 3x^2 - 3 = 0 \Rightarrow x = \pm 1, and 2y = 0 \Rightarrow y = 0. Two critical points:

(1, 0) \quad\text{and}\quad (-1, 0).

Step 3 — the second partials. Differentiate again:

f_{xx} = 6x, \qquad f_{yy} = 2, \qquad f_{xy} = 0.

Step 4 — the discriminant. D = f_{xx} f_{yy} - f_{xy}^{\,2} = (6x)(2) - 0^2 = 12x.

Step 5 — classify (1, 0). Here D = 12(1) = 12 > 0 and f_{xx} = 6(1) = 6 > 0 — so (1, 0) is a local minimum.

Step 6 — classify (-1, 0). Here D = 12(-1) = -12 < 0 — so (-1, 0) is a saddle point, regardless of f_{xx}. Along the x-axis the cubic x^3 - 3x peaks there, while along y the term y^2 climbs — up one way, down the other.

A saddle is the genuinely two-dimensional phenomenon: a mountain pass, high relative to the valleys on either side yet low relative to the peaks fore and aft. A one-variable function can't manage it — there's only one axis to disagree along. It is exactly why \nabla f = \mathbf{0} is necessary but not sufficient for an extremum.

The test above is local. To find the global maximum or minimum of f on a closed, bounded region you must check two kinds of candidate and compare their heights:

Interior critical points — solve \nabla f = \mathbf{0} inside the region.
The boundary — the extreme value may sit on the edge, where \nabla f need not vanish; restrict f to the boundary curve and optimise there (often with Lagrange multipliers).

A continuous function on a closed bounded region attains its global max and min (the extreme value theorem), so the winner is simply the largest and smallest height among all the candidates — interior and boundary alike.

Anatomy of the pure saddle

The example above buried its saddle inside a cubic. Strip the idea down to the simplest possible shape — the pure saddle — and watch exactly how "up one way, down the other" happens:

f(x, y) = x^2 - y^2.

Step 1 — the gradient. f_x = 2x and f_y = -2y, so the only critical point is the origin (0, 0).

Step 2 — the second partials and the discriminant. f_{xx} = 2, f_{yy} = -2, f_{xy} = 0, so D = (2)(-2) - 0^2 = -4 < 0 — a saddle, confirmed by the test.

Step 3 — see it directly, one slice at a time. Freeze y = 0: the slice is f(x, 0) = x^2, an upward parabola with a minimum at the origin. Now freeze x = 0: the slice is f(0, y) = -y^2, a downward parabola with a maximum at the origin. Same point, two perpendicular slices, two opposite verdicts — that disagreement is the saddle.

A bonus detail. Slice along the diagonal y = x instead: f(x, x) = x^2 - x^2 = 0 for every x — perfectly flat, a level "ridge line" running straight through the saddle. Every pure saddle hides two such special directions (here the diagonals) where the surface neither rises nor falls to first order, sandwiched between the rising and falling directions on either side.

Worked example: least material for an open box

A box factory wants an open-top box (no lid) holding a fixed volume V = 4 using as little sheet metal as possible. Let the base be x \times y and the height z.

Step 1 — eliminate the constraint. Volume fixes xyz = 4, so z = 4/(xy). Substituting turns the surface area — base plus four walls, no lid — into a function of x and y alone:

S(x, y) = xy + 2xz + 2yz = xy + \frac{8}{y} + \frac{8}{x}.

Step 2 — the gradient.

S_x = y - \frac{8}{x^2}, \qquad S_y = x - \frac{8}{y^2}.

Step 3 — solve \nabla S = \mathbf{0}. By symmetry try x = y: then x = 8/x^2 \Rightarrow x^3 = 8 \Rightarrow x = 2, so (x, y) = (2, 2), giving z = 4/(2 \cdot 2) = 1.

Step 4 — the second-derivative test. S_{xx} = 16/x^3, S_{yy} = 16/y^3, S_{xy} = 1. At (2, 2): S_{xx} = S_{yy} = 2, so D = (2)(2) - 1^2 = 3 > 0 and S_{xx} = 2 > 0 — a genuine local minimum.

Step 5 — the answer. The cheapest box has a square 2 \times 2 base and height 1, using S = (2)(2) + 8/2 + 8/2 = 4 + 4 + 4 = 12 square units of metal — and no other shape holding the same volume uses less.

D = 0 is silent, not friendly. It doesn't mean "extremum" or "saddle" — it means the second-order test has nothing to say, and you must look closer by hand (compare slices, or check higher-order terms). f = x^4 + y^4 has D = 0 at the origin, yet the origin is obviously the global minimum since f \geq 0 everywhere.
f_{xy} enters the discriminant squared. Forget the square, or flip its sign by mistake, and a genuine extremum can masquerade as a saddle or vice versa. Always compute D = f_{xx}f_{yy} - f_{xy}^2 term by term rather than trusting a half-remembered shortcut.
Critical points are only local. Solving \nabla f = \mathbf{0} inside a region never sees the boundary — the true global max or min can sit on the edge instead, where the gradient need not vanish at all.

A mountain pass is a saddle point of elevation, and its two directions tell two different stories. Walk along the ridge — the line linking the peak behind the pass to the peak ahead — and the pass is a local minimum: lower than either summit, which is exactly why it's the sensible place to cross rather than climbing all the way over a peak. Walk along the road instead — the route climbing up from one valley and back down into the next — and the pass is a local maximum: the highest point you're forced to reach on that journey.

That double identity is why passes, not peaks, decide where trade routes, railways and highways run: a pass is the lowest possible crossing of a mountain range, even though it still feels like the top of your climb. (The same shape shows up far from any mountain — a training algorithm sliding downhill on a bumpy loss landscape can get stuck for a long time near a saddle, flat in every direction it happens to check first, before finding the one direction that finally slopes away.)

Three kinds of flat point, one map

Here is the separable surface f(x, y) = (x^3 - 3x) + (y^3 - 3y), whose contours are drawn as faint rings. It has four critical points — a local min, a local max, and two saddles. Pick a critical point with the buttons: the marker jumps to it, and the panel reports its discriminant D = f_{xx} f_{yy} - f_{xy}^2 and the verdict from the test. Notice the contours close into loops around the min and max, but cross in an X at each saddle.

The saddle, in three dimensions

The pure saddle f(x, y) = x^2 - y^2 is best seen. Here it is as a rotatable surface — drag it to spin the shape into view. The origin (marked) is a genuine saddle: a minimum along one axis, a maximum along the other. That is why a zero gradient is not always a peak or a pit — the surface can rise one way and fall the other at the very same flat point.