The Bellman equation was a backward recursion over discrete stages. Let the stage shrink to an infinitesimal slice of time and that recursion becomes a partial differential equation for the value function — the Hamilton–Jacobi–Bellman (HJB) equation. It is the continuous-time crown of dynamic programming, and remarkably, the Hamiltonian steps straight back out of it.

Deriving HJB from the principle of optimality

Let V(x, t) be the cost-to-go for the system \dot{x} = f(x, u) with running cost L(x, u) and terminal cost \varphi(x(T)).

Step 1 — split the horizon at a tiny interval. The principle of optimality says the best cost from (x, t) equals the cost over [t, t+dt] plus the best cost from wherever you land:

Step 2 — Taylor-expand the cost-to-go at the next instant. Expand V about (x, t) to first order, using the partial derivatives V_t and the gradient V_x = \nabla_x V:

Step 3 — substitute and cancel V(x, t). The term V(x, t) and the time-derivative term V_t\,dt do not depend on u, so they come out of the minimisation:

Cancelling V(x, t) from both sides leaves

Step 4 — divide by dt and let dt \to 0. The o(dt) term vanishes and the bracket's common dt factors out:

This is the Hamilton–Jacobi–Bellman equation, a first-order partial differential equation for V(x, t), closed by the terminal condition V(x, T) = \varphi(x) (at the final time there is nothing left to do).

The Hamiltonian, reappearing

Look at the bracket being minimised. With L + \lambda^\top f the very definition of the Hamiltonian H(x, u, \lambda), the HJB bracket is the Hamiltonian — provided we identify the costate with the gradient of the value function:

So the costate \lambda of the maximum principle is not an abstract multiplier after all: it is the sensitivity of the optimal cost to the state, \partial V/\partial x — the shadow price of being where you are. HJB and Pontryagin are looking at the same Hamiltonian from two directions.

The two are complementary. HJB is a sufficient condition: a smooth V solving the PDE delivers the globally optimal feedback control u^\*(x, t) at every state. The maximum principle gives necessary conditions — coupled ODEs along a single optimal trajectory. One PDE for all states, versus ODEs for one path.

A worked HJB solution

Take the scalar system \dot{x} = u with the infinite-horizon cost \int_0^\infty (x^2 + u^2)\,dt, so L = x^2 + u^2 and f = u. With no explicit time dependence the value function is stationary (V_t = 0) and HJB reads

Step 1 — minimise the bracket over u. It is a quadratic in u; setting its derivative to zero, 2u + V'(x) = 0, gives the feedback u^\* = -\tfrac12 V'(x).

Step 2 — substitute back. Putting u^\* into the bracket,

Step 3 — solve. Taking the root that makes V a genuine (positive, increasing-away-from-zero) cost, V'(x) = 2x, so

The value function is the parabola V(x) = x^2 and the optimal policy is the linear feedback u^\*(x) = -x: steer toward the origin in proportion to how far away you are. No trajectory needed — HJB hands back the control law for every state at once.

The value landscape and the feedback law

Below are the two objects HJB produced: the value function V(x) = x^2 — the cost landscape, lowest at the goal x = 0 — and the feedback law u^\*(x) = -x. Slide the state x: the markers read off the cost-to-go and the control HJB prescribes there. Notice the optimal control is largest, and points hardest toward the origin, exactly where the landscape is steepest — because u^\* is set by the slope V'(x).