Every time a language model finishes a sentence, a self-driving car steadies its lane, or a recommender guesses your next film, a machine somewhere has already taken billions of derivatives to get there. Not derivatives of one variable with respect to another — derivatives of a single number (a "loss", how wrong the model is) with respect to a whole vector of millions of tunable weights, all at once. The bookkeeping that makes this possible, and keeps it sane, is matrix calculus: differentiating with respect to vectors and matrices instead of scalars.

The good news is that there is almost no new calculus here. You already know how to take a gradient and how a Jacobian packages many partial derivatives into a matrix. Matrix calculus is mostly a change of language: instead of writing out n partial-derivative equations, you write one tidy line with vectors and matrices in it — and then you learn the handful of identities that let you differentiate those lines by sight. Master the cheat sheet below and you can differentiate a neural network's loss on the back of a napkin.

Three objects: gradient, Jacobian, Hessian

Everything in matrix calculus is built from three arrangements of partial derivatives. Which one you get depends only on what goes in and what comes out.

1. The gradient — scalar out, vector in. For a scalar function f(\mathbf{x}) of a vector \mathbf{x} = (x_1, \dots, x_n), the derivative \partial f / \partial \mathbf{x} is the vector that collects one partial per input:

Same object you already met — just written as "the derivative of a scalar with respect to a vector". It has one number for every input direction.

2. The Jacobian — vector out, vector in. For a vector-valued function \mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m (so m outputs, n inputs), the derivative is the m \times n matrix whose row i is the gradient of the i-th output:

Stack m gradients as rows and you have the Jacobian. A gradient is just the special case m = 1: a Jacobian with a single row.

3. The Hessian — scalar out, differentiated twice. Take a scalar f(\mathbf{x}) and differentiate again: the matrix of all second partials is the Hessian, the Jacobian of the gradient,

It measures curvature — how the gradient itself bends — and (for nice functions) it is symmetric, because mixed partials commute: \partial^2 f / \partial x_i \partial x_j = \partial^2 f / \partial x_j \partial x_i.

The cheat sheet every ML practitioner memorises

Once functions are written with vectors and matrices, a small set of identities lets you differentiate them without ever peeking at individual components. Treat these like the power rule of matrix calculus — worth knowing cold. Throughout, \mathbf{a} is a constant vector, A a constant matrix, and \mathbf{x} the variable vector.

A quick sanity check on the linear form. Write \mathbf{a}^{\top}\mathbf{x} = a_1 x_1 + a_2 x_2 + \dots + a_n x_n. Differentiating with respect to x_k kills every term except a_k x_k, leaving a_k. Collect those and the gradient is (a_1, \dots, a_n) = \mathbf{a}. Every identity above can be checked the same way — write out one component, differentiate, and read the pattern.

Worked example 1 — the least-squares loss (where the identities pay off)

This is the calculation of classical machine learning, and it uses three of the identities at once. We fit a linear model to data by minimising the squared error. With data matrix A, unknown weights \mathbf{x}, and targets \mathbf{b}, the loss is the squared length of the residual:

Step 1 — expand the product. Multiply it out, remembering that transposing a product reverses order, (A\mathbf{x})^{\top} = \mathbf{x}^{\top} A^{\top}:

The two middle terms are each a single number, and a number equals its own transpose, so \mathbf{b}^{\top} A\,\mathbf{x} = (\mathbf{b}^{\top} A\,\mathbf{x})^{\top} = \mathbf{x}^{\top} A^{\top}\mathbf{b}. They are the same thing; combine them:

Step 2 — differentiate term by term with the cheat sheet. The matrix A^{\top} A is symmetric (indeed (A^{\top}A)^{\top} = A^{\top} A), so the quadratic-form rule gives 2 A^{\top} A\,\mathbf{x}. The linear form \mathbf{a}^{\top}\mathbf{x} with \mathbf{a} = A^{\top}\mathbf{b} differentiates to \mathbf{a}, and the constant vanishes:

Step 3 — set the gradient to zero. The loss is a bowl (a positive quadratic), so its one minimum is where \nabla f = \mathbf{0}:

Those are the famous normal equations of linear regression — the closed-form best fit — and they dropped out of three lines of matrix calculus. No summation signs, no writing out components: the identities did the work.

Worked example 2 — a concrete quadratic form

Let us make \nabla(\mathbf{x}^{\top}\! A\,\mathbf{x}) = (A + A^{\top})\mathbf{x} completely concrete with a small symmetric A. Take

Step 1 — write out the scalar. Multiplying \mathbf{x}^{\top}\! A\,\mathbf{x} gives a single number:

(The cross term is 2x_1 x_2 because the off-diagonal 1 appears twice, once above and once below the diagonal.)

Step 2 — take the two partials directly.

Step 3 — check against the identity. Since A is symmetric, A + A^{\top} = 2A, so the rule predicts

Identical. And notice the Hessian here is exactly that constant matrix 2A = \begin{bmatrix} 4 & 2 \\ 2 & 6 \end{bmatrix} — for a quadratic, the curvature is the same everywhere, which is why a quadratic bowl has one clean minimum.

Why any of this matters: gradient descent and backpropagation

Training a model is an optimisation problem: adjust the weights \mathbf{x} to make a scalar loss f(\mathbf{x}) as small as possible. Because -\nabla f points in the direction of steepest descent, the recipe is the same one line whether \mathbf{x} has two components or two billion:

with a small step size \eta (the "learning rate"). Every training step needs \nabla f — the gradient of a loss built from matrix and vector expressions — so the identities on this page are precisely what makes a training step computable. The least-squares gradient 2A^{\top}(A\mathbf{x} - \mathbf{b}) we just derived is the simplest real example of one.

A deep network is a stack of functions, layer after layer, so its loss is a big composition f = f_L \circ \dots \circ f_2 \circ f_1. Differentiating a composition is the job of the vector chain rule: the overall Jacobian is the product of each layer's Jacobian. Backpropagation is nothing more than evaluating that product efficiently — multiplying the layer Jacobians from the output end back toward the input, reusing work as it goes. So the entire modern edifice of deep learning rests on two things from this page: the gradient of a scalar loss, and the chain rule for Jacobians.

See the gradient field of a loss

Here is the loss f(x, y) = x^2 + 2y^2 as a contour map — the rings are its level curves, tighter where the bowl is steep. The short arrows form the gradient field: at every point, \nabla f = (2x,\, 4y) points in the direction of steepest ascent, always crossing the contour at a right angle and growing longer where the walls are steeper. Training steps the opposite way, along -\nabla f, straight toward the minimum at the centre.

Drag the sliders to move the marked point P around and watch its gradient arrow (the bold one) swing to point uphill. That single arrow is exactly what gradient descent reads off before every step — flip its sign and you have the direction the optimiser actually moves.