Conditioning

Every measurement you feed into a computation is a little bit wrong — a rounded decimal, a noisy sensor, a truncated constant. The question that decides whether your answer is trustworthy is blunt: how much does this problem amplify those errors? Feed in data that is off by 1%, and does the answer come out off by 1%… or off by 10000%?

Remarkably, a single number answers that question for any linear system. It is the numerical danger rating of the problem — small means noise stays small and you are safe; huge means a whisper of input error explodes into a scream of output error. That number is the condition number, and learning to read it is the difference between a result you can stake a bridge on and a result that is pure fiction.

How much does the answer wobble when the data wobbles? That sensitivity is conditioning, and for a linear problem it is captured by a single number built from the singular values — the condition number

\kappa(G) = \frac{\sigma_{\max}}{\sigma_{\min}}.

It is the ratio of the largest singular value to the smallest. Those singular values measure how much the operator stretches (\sigma_{\max}) and how much it squashes (\sigma_{\min}) different directions. When the smallest is a whisker above zero, dividing by it during inversion is where the damage is done.

The condition number bounds how a relative error in the data becomes a relative error in the solution:

\frac{\|\delta m\|}{\|m\|} \le \kappa(G)\,\frac{\|\delta d\|}{\|d\|}.

A small \kappa (near 1) is well-conditioned: errors stay the same size. A huge \kappa is ill-conditioned: a 0.1% data error can become a 100% solution error.

The ellipse aspect ratio

Picture the forward map sending the unit circle to an ellipse: its longest semi-axis is \sigma_{\max}, its shortest \sigma_{\min}, and \kappa is the aspect ratio. A near-circular ellipse (\kappa \approx 1) is easy to invert; a long thin sliver (\kappa \gg 1) squashes one direction almost to nothing, and undoing that squash magnifies any error along it. Conditioning is a property of the operator, set before any noise enters — it tells you how much trouble to expect.

Drag \sigma_{\min} down toward zero and watch the ellipse collapse into a needle. That needle is the picture of an ill-conditioned problem: the map has thrown away almost all information about one direction, so no amount of arithmetic can honestly recover it.

Worked example — reading a condition number off the singular values

Suppose a small imaging operator has singular values \sigma_1 = 8,\ \sigma_2 = 2,\ \sigma_3 = 0.004. The largest is 8, the smallest is 0.004, so

\kappa = \frac{\sigma_{\max}}{\sigma_{\min}} = \frac{8}{0.004} = 2000.

Only the extremes matter — the middle value \sigma_2 = 2 plays no part at all. A single tiny singular value at the bottom is enough to poison the whole problem, however healthy the rest look.

Worked example — turning κ into "digits lost"

Here is the rule of thumb that makes the condition number visceral. A condition number of 10^p means you can lose about p decimal digits of accuracy. So \kappa = 10^6 costs you roughly six digits:

\text{output error} \approx \kappa \times \text{input error} = 10^6 \times 0.01 = 10^4 = 10000\%.

Read that again — a 1% wobble in the data can be blown up into a 10000% wobble in the answer. The first six significant figures of your result are, in the worst case, meaningless. If you started with 15 good digits (double precision), you walk away with about 9.

Worked example — well-conditioned vs nearly singular, side by side

Contrast two matrices. The first,

A = \begin{pmatrix} 3 & 0 \\ 0 & 2 \end{pmatrix}, \qquad \kappa(A) = \frac{3}{2} = 1.5,

is beautifully well-conditioned: a 1% data error stays a roughly 1.5% answer error. Now the second,

B = \begin{pmatrix} 1 & 1 \\ 1 & 1.0001 \end{pmatrix}, \qquad \kappa(B) \approx 4\times 10^{4},

has two almost identical rows, so it is almost singular. Its singular values are about 2 and 5\times 10^{-5}, giving \kappa \approx 40000. Solving Bx = d can turn a rounding error in the fifth decimal place of d into a completely different x. Same size of matrix, wildly different trustworthiness — and you would never know without looking at \kappa.

The single most common misconception: people meet a wrong answer, blame their algorithm, and go hunting for a cleverer solver. But the condition number is a property of the problem — the matrix A itself — not of the method you use to attack it. No solver, however ingenious, can beat it.

If \kappa(A) is enormous, the reliable information you are asking for simply isn't in the data. A better factorization, higher precision, a fancier iteration — none of them can create information that the measurement threw away. The only real fixes change the problem: add regularization to stabilize it, or go back and take better, more independent measurements. Hunting for a magic solver is the beginner's trap; recognising that the problem itself must change is the expert's move.

The condition number explains why some computations are hopeless no matter how powerful your machine. Double-precision arithmetic carries about 16 digits, so a matrix with \kappa \approx 10^{16} eats every last one — your answer is essentially random digits, all signal drowned by rounding.

Real disasters trace back to exactly this. The Vancouver Stock Exchange index (1982) was recomputed thousands of times a day, each time truncating rather than rounding; the tiny errors accumulated until the index had silently drifted to about half its true value. The Patriot missile failure at Dhahran (1991) came from a clock counted in tenths of a second — a number with no exact binary form — whose rounding error grew over 100 hours of uptime until the interceptor mistimed an incoming Scud by half a kilometre. Neither was a hardware fault. Both were conditioning and error accumulation, ignored.

Not really — \kappa \ge 1 always (the largest singular value can't be smaller than the smallest), and \kappa = 1 is the dream: an orthogonal map, a perfect rotation or reflection that preserves every length. Those are the safest problems in all of numerical computing. The catch is that most interesting inverse problems — deblurring an image, imaging the Earth's interior, reconstructing a CT scan — are ill-conditioned by nature, because gentle physics smooths away fine detail on the way out, and undoing that smoothing is exactly the dangerous, error-amplifying step.