Lagrange Multipliers

Picture an actual fence. You are standing in a field bounded by a curved wire fence, and you want to find the sunniest spot — but you are only allowed to walk along the wire itself, never inside it. The sunniest spot in the whole field might sit well inside the boundary, completely out of reach; fence-bound, the best you can do is find the sunniest point on the fence. That everyday picture is exactly constrained optimisation, and it is everywhere serious: maximise a factory's output on a fixed budget, minimise a can's surface area for a fixed volume, find the point on a road closest to your house.

Unconstrained optimisation sets the whole gradient to zero — that's the free peak with no fence around it at all. But the interesting problems come fenced in: maximise output subject to a fixed budget, minimise surface area at a fixed volume. We want to optimise f(x, y) while a constraint holds,

g(x, y) = c.

Now \nabla f = \mathbf{0} is the wrong condition: the best point on the constraint curve is rarely a free peak of f. The right condition is startlingly clean. At a constrained optimum the two gradients are parallel:

\nabla f = \lambda\, \nabla g,

for some scalar \lambda, the Lagrange multiplier. One little equation captures the whole geometry.

Deriving the condition geometrically

The argument rests entirely on what the gradient means — steepest ascent, and perpendicular to level curves.

Step 1 — you are confined to the constraint curve. Every allowed point satisfies g = c, so you may only travel along the curve g(x, y) = c. Let \mathbf{T} be its tangent direction at a candidate point.

Step 2 — at a constrained max, moving can't increase f. If stepping along the curve in direction \mathbf{T} raised f even slightly, the point wouldn't be the maximum — you'd just take that step. So the rate of change of f along the curve must be zero. That rate is a directional derivative:

D_{\mathbf{T}} f = \nabla f \cdot \mathbf{T} = 0.

Step 3 — so \nabla f has no component along the constraint. A zero dot product means \nabla f is perpendicular to the tangent \mathbf{T} of the constraint curve.

Step 4 — but \nabla g is also perpendicular to that tangent. The constraint curve is itself a level curve of g (the level g = c), and a gradient is always perpendicular to its own level curve:

\nabla g \cdot \mathbf{T} = 0.

Step 5 — two vectors perpendicular to the same line are parallel. In the plane there is only one direction perpendicular to \mathbf{T} (up to sign). Both \nabla f and \nabla g point along it, so they are scalar multiples of each other:

\nabla f = \lambda\, \nabla g.

Geometrically: at the optimum the level curve of f is tangent to the constraint curve. The contours of f are sliding across the constraint, and the last one they touch before leaving is the one that just kisses it — sharing a tangent line, hence parallel normals.

To find the extrema of f(x, y) subject to g(x, y) = c (with \nabla g \neq \mathbf{0} on the constraint), solve the system:

Parallel gradients: \nabla f = \lambda\, \nabla g, i.e. f_x = \lambda g_x and f_y = \lambda g_y.
The constraint itself: g(x, y) = c.
That is three equations in the three unknowns x, y, \lambda; solve for the candidate points.
Evaluate f at every solution; the largest value is the constrained maximum, the smallest the constrained minimum.

A worked example

Maximise f(x, y) = xy subject to the constraint x + y = 10. (The largest-area rectangle of a fixed perimeter, in disguise.)

Step 1 — the two gradients. With g(x, y) = x + y,

\nabla f = (y,\, x), \qquad \nabla g = (1,\, 1).

Step 2 — set them parallel. \nabla f = \lambda \nabla g gives two equations:

y = \lambda \cdot 1, \qquad x = \lambda \cdot 1.

Step 3 — eliminate \lambda. Both equal \lambda, so x = y.

Step 4 — use the constraint. Substitute x = y into x + y = 10:

x + x = 10 \;\Rightarrow\; x = 5, \quad y = 5.

Step 5 — the optimal value. f(5, 5) = 5 \cdot 5 = 25, with multiplier \lambda = 5. Among all ways to split 10 into two parts, the product is largest when the parts are equal — the constrained maximum is 25.

The multiplier \lambda looks like algebraic scaffolding to be discarded, but it carries real meaning. It measures how much the optimal value of f would change if you relaxed the constraint by one unit. Writing the optimum as a function of the constraint level c,

\frac{d f^{*}}{d c} = \lambda.

In our example \lambda = 5: nudging the budget from x + y = 10 to 11 would lift the maximum product by about 5 (indeed the new optimum is 5.5^2 = 30.25, up 5.25 — and 5 is the first-order estimate). To an economist this is a shadow price: the marginal value of one more unit of the scarce resource. It tells a firm exactly how much it should be willing to pay to loosen a binding constraint — the sensitivity of the prize to the fence.

Second worked example — a circular fence, and both signs of λ

Maximise (and minimise!) f(x, y) = xy subject to x^2 + y^2 = 8 — a circular fence this time, of radius 2\sqrt{2}. Unlike the straight fence above, a closed loop has no endpoints to worry about, but it does have two kinds of turning point on it: a highest value and a lowest value of f. Honestly working the system all the way through, without discarding anything early, is the whole point of this example.

Step 1 — the gradients. With g(x, y) = x^2 + y^2,

\nabla f = (y,\, x), \qquad \nabla g = (2x,\, 2y).

Step 2 — the parallel-gradient equations. \nabla f = \lambda \nabla g gives

y = 2\lambda x, \qquad x = 2\lambda y.

Step 3 — eliminate \lambda without dividing by anything that might be zero. Multiply the first equation by y and the second by x:

y^2 = 2\lambda xy, \qquad x^2 = 2\lambda xy \;\Rightarrow\; x^2 = y^2 \;\Rightarrow\; y = \pm x.

Step 4 — bring in the constraint for each branch. Two branches, two families of solution.

y = x: the constraint becomes 2x^2 = 8, so x = \pm 2, giving the points (2, 2) and (-2, -2). Plugging back into y = 2\lambda x (with y = x \neq 0) gives \lambda = \tfrac12.
y = -x: the constraint becomes 2x^2 = 8 again, so x = \pm 2, giving (2, -2) and (-2, 2). Now y = 2\lambda x with y = -x forces \lambda = -\tfrac12.

Step 5 — classify by comparing f, not by inspecting \lambda. Four candidates, four values of f = xy:

f(2,2) = 4, \quad f(-2,-2) = 4, \quad f(2,-2) = -4, \quad f(-2,2) = -4.

So the constrained maximum is 4, reached at (2,2) and (-2,-2) — both with the positive multiplier \lambda = \tfrac12 — and the constrained minimum is -4, reached at (2,-2) and (-2,2), both with the negative multiplier \lambda = -\tfrac12. Nothing about the sign of \lambda told us which pair was which — only comparing the actual values of f did.

Third worked example — a budget, and λ as a shadow price

A workshop turns two inputs, x units of steel and y units of skilled labour, into output Q(x, y) = xy. Steel costs \$4 a unit and labour costs \$2 a unit, and the month's budget is a fixed \$40:

g(x, y) = 4x + 2y = 40.

Step 1 — the gradients. \nabla f = (y, x) and \nabla g = (4, 2).

Step 2 — parallel-gradient equations. \nabla f = \lambda \nabla g gives y = 4\lambda and x = 2\lambda, so y = 2x.

Step 3 — solve with the budget. Substituting y = 2x into 4x + 2y = 40 gives 4x + 4x = 40, so x = 5, y = 10, and \lambda = x/2 = 2.5.

Step 4 — read \lambda as a price. The optimal output is Q(5, 10) = 50. Because \lambda = 2.5, loosening the budget by one more dollar — to \$41 — should lift the best-achievable output by about 2.5 units, without re-solving the whole system. That's the shadow price at work: it turns a re-optimisation into a single multiplication.

The system \nabla f = \lambda \nabla g plus the constraint is a recipe for candidates, not automatically for the answer. Three traps:

Never discard a candidate because \lambda is negative or zero. In the circular-fence example above, the minimum came with \lambda = -\tfrac12 and the maximum with \lambda = +\tfrac12 — the sign carried no information about which was which. \lambda = 0 is fine too: it just means the constraint happens not to bind the value of f at that point.
Solving the equations is not the last step. Every solution of the system is only a candidate extremum. To classify them, plug every candidate back into f and compare — exactly as the circle example did with four points and two values.
Check the fine print. The method assumes \nabla g \neq \mathbf{0} everywhere on the constraint — a point where the constraint curve has no well-defined tangent is a blind spot the method can miss. And if the constraint set has endpoints (say, it's only defined for x in a closed interval rather than running all the way round a loop), those endpoints must be checked separately — tangency is a condition for an interior optimum on the curve, and an endpoint has no interior on one side.

Confusingly, "Lagrange" also names something else entirely in space: the five Lagrange points of a two-body system (like the Sun and the Earth), spots where the gravitational tug of both bodies and the orbital motion balance out so a tiny third object can sit there almost motionless relative to the other two. The James Webb Space Telescope famously parks at L_2, 1.5 million kilometres beyond Earth, riding along in lockstep with our planet's orbit forever.

It's the same eighteenth-century mathematician, Joseph-Louis Lagrange — a man who seemingly couldn't stop finding balance points, whether the "forces" involved were literal gravity or the abstract pull of a function against a constraint. But the multiplier \lambda from this page and the points L_1, \ldots, L_5 from orbital mechanics share nothing but a surname — a fun trap for anyone skimming search results.

Slide to the tangency

The straight line is the constraint x + y = 10; the curved arcs are level curves of f = xy (hyperbolae, one per value of the product). Slide the point along the constraint. Watch the contour through the point: almost everywhere it cuts across the line, meaning you could slide to a higher contour. At x = y = 5 the contour is tangent — its gradient \nabla f = (y, x) lines up with \nabla g = (1, 1) — and the product f = xy peaks at 25.

The objective as a rotatable surface