A rocket must reach orbit on the least fuel. A robot arm must swing to a target in the least time. A power grid, a portfolio, an anaesthetic drip, a self-driving car — each is a system we can nudge, and for each there is a best way to nudge it. Optimal control is the mathematics of steering a dynamical system to minimise a cost — the theory that answers, rigorously, "what is the best possible way to drive this thing?"

It is one of applied mathematics' crown jewels, fusing differential equations (how the system moves), the calculus of variations (optimising over whole trajectories), and linear algebra (the state-space machinery) into a single, powerful toolkit — and it is the direct ancestor of modern reinforcement learning.

Two great ideas, one answer

The whole subject is held up by two towering results that turn out to be two views of the same truth. Pontryagin's maximum principle treats the best trajectory as a stationary point of a cost and produces a set of necessary conditions — a Hamiltonian and a backward-running "costate". Dynamic programming instead defines the best cost-to-go from every state and derives the Hamilton–Jacobi–Bellman equation. One is local and trajectory-based, the other global and feedback-based; reconciling them — the costate is the gradient of the value function — is one of the most satisfying moments in all of applied maths.

The shape of the journey

Six stages, from a single controlled trajectory to stochastic feedback and reinforcement learning.

• Stage 1 — Foundations. State-space models, the cost functional, and whether a system can even be steered (controllability) or seen (observability).
• Stage 2 — Pontryagin's maximum principle. The Hamiltonian, the costate, and the necessary conditions for an optimal trajectory — including bang-bang and time-optimal control.
• Stage 3 — Dynamic programming. Bellman's principle, the value function, and the Hamilton–Jacobi–Bellman equation — reconciled with the maximum principle.
• Stage 4 — The Linear-Quadratic Regulator. The one case that solves in closed form: the Riccati equation and the optimal feedback law u = −Kx.
• Stage 5 — Estimation & stochastic control. The Kalman filter, control under noise, the stochastic HJB, and LQG with the separation principle.
• Stage 6 — Modern & numerical. Model-predictive control, numerical methods, and the bridge from HJB to reinforcement learning.

Stage 1 — Foundations

Describe the system, the cost, and what is even possible.

1. What Is Optimal Control?
2. State-Space Models
3. The Cost Functional
4. Linear Systems and the Matrix Exponential
5. Controllability
6. Observability

Stage 2 — Pontryagin's maximum principle

The necessary conditions an optimal trajectory must satisfy.

1. The Hamiltonian and Costate
2. The Maximum Principle
3. Deriving the Maximum Principle
4. Bang-Bang Control
5. Time-Optimal Control

Stage 3 — Dynamic programming

The value function, the HJB equation, and how it relates to Pontryagin.

1. The Principle of Optimality
2. The Bellman Equation
3. Value and Policy Iteration
4. The Hamilton–Jacobi–Bellman Equation
5. Maximum Principle vs Dynamic Programming

Stage 4 — The Linear-Quadratic Regulator

The case that solves in closed form — the workhorse of real control engineering.

1. The LQ Problem
2. The Riccati Equation
3. The Algebraic Riccati Equation
4. The Optimal Feedback Law
5. The Inverted Pendulum

Stage 5 — Estimation & stochastic control

Real systems are noisy and only partly seen — estimate the state, then control it.

1. State Estimation
2. The Kalman Filter
3. Stochastic Optimal Control
4. The Stochastic HJB Equation
5. LQG and the Separation Principle

Stage 6 — Modern & numerical

How optimal control is actually computed today — and where it meets machine learning.

1. Model-Predictive Control
2. Numerical Optimal Control
3. Optimal Control and Reinforcement Learning

Let's get started

We begin by framing the central question precisely: a system that evolves, a knob we can turn, and a cost we want to make as small as possible.

Let's get started → What Is Optimal Control?