The Linear Inverse Problem

Here is a piece of good news that borders on the miraculous. A CT scanner, a radio telescope deblurring a smeared star, and a survey ship listening for oil beneath the seabed are attacking wildly different physics — X-rays, light, sound. Yet an enormous fraction of them boil down to the same three-letter equation:

A\,x = b.

Here A is the known forward operator (the physics, written as a matrix), b is your measured data, and x is the hidden thing you want to recover — the tissue density, the true image, the rock structure. "Invert the physics" has become "solve a linear system." That translation is the single most powerful move in the whole field: master Ax=b and you hold a skeleton key to dozens of sciences at once.

The setup

An enormous share of inverse problems are linear: the data depends linearly on the model. Then the forward map is a matrix, and the whole problem is the system

d = G\,m,

with m \in \mathbb{R}^n the unknown model, d \in \mathbb{R}^M the data, and G the M \times n forward operator. Each row of G is one measurement — a recipe that weights the model parameters and reports a number. Inverting the problem means solving this system for m.

(The letters differ by tradition — physicists write d = Gm, numerical analysts write Ax = b — but it is one and the same equation: A = G the operator, x = m the unknown, b = d the data.) A row of G that reads [\,0\;1\;1\;0\,] simply says "my instrument reports the sum of the second and third model parameters" — a matrix-times-vector dotted against the unknown.

Worked example: a tiny tomography

Imagine a 2\times 2 grid of four unknown densities x_1, x_2, x_3, x_4, and suppose each measurement is the total density along one row or column of the grid. Two horizontal beams and two vertical beams give four numbers:

x_1 + x_2 = b_1,\quad x_3 + x_4 = b_2,\quad x_1 + x_3 = b_3,\quad x_2 + x_4 = b_4.

Stacked up, that is exactly Ax = b with a known 4\times 4 matrix

A = \begin{bmatrix} 1&1&0&0\\ 0&0&1&1\\ 1&0&1&0\\ 0&1&0&1 \end{bmatrix},\qquad x = \begin{bmatrix}x_1\\x_2\\x_3\\x_4\end{bmatrix}.

This little A is a cartoon CT scanner. And it already bites: the four equations are not independent (the two row-sums and the two column-sums must add to the same grand total), so A is singular — you cannot simply write x = A^{-1}b. Add a beam along a diagonal and you break the tie. Real scanners fire beams from hundreds of angles for exactly this reason.

Why you often cannot just compute x = A^{-1}b: three different failures show up constantly.

Non-square. With more measurements than unknowns, or fewer, A is not even a square matrix, and A^{-1} does not exist.
Singular. Even a square A can have dependent rows (like the grid above), so its inverse does not exist.
Nearly singular. Worst of the three: A^{-1} exists but is so unstable that noise wrecks the answer — the next section.

The non-square cases are where the least-squares solution and the generalized inverse step in to give a sensible answer even when a true inverse does not exist.

Three shapes of problem

The shape of G already tells you what to expect:

Even-determined (M = n, G invertible): a unique solution m = G^{-1}d — the rare, easy case.
Over-determined (M > n): more measurements than unknowns. Usually no exact solution because the data is noisy and inconsistent — we seek a best fit (least squares). Think of fitting a straight line to twenty scattered points: two points fix a line, so eighteen extras over-constrain it and no line hits them all.
Under-determined (M < n): fewer measurements than unknowns. Infinitely many models fit — we must choose among them (minimum norm, or a prior). One equation x_1 + x_2 = 5 in two unknowns already has a whole line of solutions.

Real problems are often both at once — over-determined in some directions, under-determined in others — which is exactly what the SVD is built to untangle.

Linear forward model: d = Gm, with each row of G a single measurement.
M = n even-determined; M > n over-determined (best fit); M < n under-determined (many fits).
Rank, not just shape, decides solvability — and most real problems mix all three behaviours.

See the instability: near-parallel lines

A 2\times 2 system Ax=b is two lines in the plane; the solution x is where they cross. When the two equations are nearly the same — nearly parallel lines — A is close to singular. Nudge the data b by a whisker with the slider and watch the crossing point shoot across the plane. That runaway sensitivity is ill-conditioning, the beating heart of what makes inverse problems hard.

The most dangerous trap in the whole subject: even when A is technically invertible, the naive x = A^{-1}b can be catastrophically wrong.

If A is ill-conditioned — nearly singular — then A^{-1} contains gigantic numbers, and it multiplies the tiny measurement noise in b up into an enormous error in x. The arithmetic is flawless; the answer is garbage. A change in the data too small to see can flip the reconstruction from a healthy scan to a fake tumour. This is not a rare pathology — it is the generic behaviour of the forward maps that arise in imaging and geophysics, because smoothing physics makes near-parallel equations. It is the reason the naive inverse is almost never used in practice, and the reason Tikhonov regularization and its cousins exist: they deliberately trade a little bias for a huge gain in stability.

It is genuinely strange how far Ax=b reaches. Fill A with X-ray path lengths and it becomes a CT scanner. Fill it with a blur kernel and it deblurs a telescope image — famously, in the 1990s the Hubble Space Telescope launched with a mirror ground to the wrong shape by a smear the width of a human hair, and for three years astronomers rescued its blurred pictures by solving exactly this kind of linear inverse problem in software before a corrective optic could be flown up. Fill A with how seismic waves travel through rock and the same machine infers oil and gas deposits a mile beneath the seabed.

Different A, different b, but one algorithm. Learn to solve the linear inverse problem well — stably, against noise — and you have not learned one trick, you have learned the trick behind a whole shelf of Nobel-adjacent technologies. That is why this humble equation is worth taking so seriously.