Why Inverse Problems Are Hard

A CT scanner never actually sees a slice of your body. It fires X-rays straight through you from hundreds of angles and measures how much each ray dims — and then a computer must work backwards from those shadows to the picture of what is inside. That backwards step is an inverse problem, and it is treacherous in a way the forward direction never is. Radar, sonar, oil prospecting, image deblurring, and reconstructing a signal from a blurry sensor are all the same shape: you measure an effect and try to recover the cause.

Why is going backwards so much harder than going forwards? Around 1900, Jacques Hadamard gave the sharpest possible answer. He said a problem is well-posed only if it passes three tests — and inverse problems have a nasty habit of failing at least one.

Hadamard's three conditions

Every question you could ask about "which cause produced this data?" must survive three checks before the answer means anything:

Existence. Is there any cause that would produce the data we measured? If your data can't be explained by any model, something is wrong — usually because real measurements carry noise and no exact cause reproduces them perfectly.
Uniqueness. Is there exactly one cause, or do many different causes fit the data equally well? If a dozen wildly different pictures all cast the same shadows, the data alone can never tell you which one is real.
Stability. If the data wobbles a tiny bit — as real measurements always do — does the answer wobble a tiny bit too, or does it lurch to something completely different? This is continuous dependence on the data, and it is the quiet assassin of the three.

A problem that passes all three is well-posed. Miss even one and it is ill-posed — and the way it fails tells you exactly what you must do to rescue it.

The three failures, in matrix terms

For the linear problem d = Gm — data d, model m, forward operator G — each Hadamard condition has a crisp linear-algebra meaning:

Existence fails when d lies outside the column space of G — no model reproduces the (noisy) data exactly. Fix: accept a best fit instead of an exact one.
Uniqueness fails when G has a non-trivial null space: any m_0 with Gm_0 = 0 can be added to a solution and the data cannot tell. Fix: pick a preferred solution (minimum norm, or a prior).
Stability fails when G has tiny singular values: inverting divides by them, so noise is amplified enormously. Fix: regularization.

The first two can be repaired by the generalized inverse. The third — instability — is the deep difficulty, and the reason this course spends so long on conditioning and regularization. (It is the same instability that made the backward heat equation hopeless.)

Existence — a solution exists (fails if d \notin column space of G).
Uniqueness — only one solution (fails if G has a null space).
Stability — continuous dependence on data (fails when G has tiny singular values).
Inverse problems are typically ill-posed; instability is the hardest failure to cure.

Worked example 1 — diagnosing a failure of uniqueness

Suppose two hidden numbers m_1 and m_2 produce a single measurement, their sum:

d = m_1 + m_2 = 10, \qquad G = \begin{bmatrix} 1 & 1 \end{bmatrix}.

Does a solution exist? Certainly — (5, 5) works. But so does (0, 10), and (-3, 13), and infinitely many others: any pair on the line m_1 + m_2 = 10. This is an under-determined system — one equation, two unknowns — and it fails uniqueness. The direction m_0 = (1, -1) satisfies Gm_0 = 0: you can slide along it forever and the data never notices. That line is the null space. No amount of better measuring of this one number will ever separate the two — you must bring in extra information (a prior, or the minimum-norm choice, which here picks the tidy answer (5, 5)).

Worked example 2 — a unique solution that is still useless

Now a system with a genuine, one-and-only solution. Two nearly-identical measurements:

G = \begin{bmatrix} 1 & 1 \\ 1 & 1.001 \end{bmatrix}, \qquad d = \begin{bmatrix} 2 \\ 2 \end{bmatrix}.

Solve it exactly and you get m = (2, 0). Existence: passed. Uniqueness: passed — G is invertible, so there is exactly one answer. Everything looks fine on paper. Now nudge the data by a hair, the kind of noise every real sensor adds:

d = \begin{bmatrix} 2 \\ 2.001 \end{bmatrix} \;\Longrightarrow\; m = \begin{bmatrix} 1 \\ 1 \end{bmatrix}.

A change of 0.001 in the data — one part in two thousand — flipped the answer from (2, 0) to (1, 1). That is a failure of stability. The two rows of G are almost parallel; geometrically we are finding where two nearly-coincident lines cross, and a whisker of noise skates that intersection point far across the plane. The exact solution exists and is unique and is worthless, because measurement noise gets multiplied into nonsense. Just how much it gets multiplied is measured by the condition number.

Drag the slider below: a well-posed problem passes data wobble through unchanged (slope 1); an ill-posed one multiplies it by a large factor, so a tiny horizontal step becomes a towering vertical one.

Stability is the subtle killer

Existence and uniqueness announce themselves loudly: no solution, or too many. You notice at once. Instability hides. The computer hands you a single, precise-looking vector of numbers and never warns you that those numbers are mostly amplified noise. This is why an inverse problem can have a beautiful closed-form solution that a careful student derives perfectly — and still produce garbage when you feed it real data. The danger isn't in the algebra; it's in the sensitivity.

That single realisation reshapes the whole subject. Instead of asking "what is the exact solution?" we learn to ask "what is a stable approximate solution that is close enough to the truth and doesn't amplify the noise?" — which is the job of regularization.

When someone calls a problem ill-posed, they almost always mean the stability condition fails — not existence and not uniqueness. This trips people up constantly. A textbook formula for the solution may exist, may be provably unique, and may be written on the board in one clean line — and still be practically worthless, because it amplifies measurement noise into meaningless static.

The moral: naïvely "solving an inverse problem exactly" is often the wrong thing to do. The exact solution is precisely the one that trusts every last digit of your noisy data, and that trust is fatal. Deliberately solving it less exactly — stabilising it — is what recovers a sensible answer. If your reconstruction looks like pure noise, don't hunt for an algebra bug; suspect instability, and reach for regularization.

When Hadamard laid down his three conditions, he believed that ill-posed problems were mathematical curiosities that could not describe real nature. Physics, he reasoned, is well-posed: a real cause produces a definite effect that depends smoothly on the conditions, so any equation modelling reality must be well-posed too. Ill-posed problems, to him, were unphysical — artefacts of sloppy modelling, not descriptions of the world.

History demolished that view. It turned out that some of the most important technologies of the twentieth century are precisely ill-posed inverse problems: CT scanning (reconstructing a body from its X-ray shadows), radar and sonar imaging, seismic exploration for oil, deblurring telescope and microscope images. These aren't unphysical at all — they are how we see inside the human body and under the ground. Engineers simply had to learn to tame ill-posedness rather than wish it away. The entire modern field of regularization exists because Hadamard's "unphysical curiosities" turned out to run half of medical imaging.