Ill-Posedness and Noise
Now we see why ill-posed problems are so dangerous. The
pseudoinverse
reconstructs the model as a sum over singular directions,
\hat m = \sum_i \frac{u_i^{\mathsf T} d}{\sigma_i}\,v_i.
Split the data into truth plus noise, d = d_{\text{true}} + e. The truth
term is fine. But the noise contributes \sum_i (u_i^{\mathsf T}e)/\sigma_i\,v_i,
and for the small singular values that 1/\sigma_i is enormous. A whisper
of noise along a weak direction becomes a roar in the reconstruction.
The high-index catastrophe
Forward operators that smooth (blurring, integration, diffusion) have singular values that
decay — the high-index directions are the fine details, and the operator barely
records them. Inverting multiplies each by 1/\sigma_i, so the
finest-detail components are amplified the most. The naïve reconstruction is dominated by violently
amplified high-frequency noise — wild oscillations that swamp the true signal.
This is the same pathology as the
backward heat equation:
undoing smoothing amplifies exactly what smoothing suppressed. The cure is to stop trusting
the smallest singular values — which is what regularization does.
Watch the reconstruction explode
The faint curve is the true solution's coefficients (decaying, well-behaved). The bold curve is
the naïve reconstruction (c_i + e)/\sigma_i with singular values
\sigma_i that decay. With the noise slider at zero they agree; add even a
little noise and the high-index tail erupts — the smallest \sigma_i
amplify it beyond all reason.
- The noise contributes \sum_i (u_i^{\mathsf T}e)/\sigma_i\,v_i to the reconstruction.
- Smoothing operators have decaying \sigma_i, so high-index (fine-detail) noise is amplified most.
- Naïve inversion is dominated by amplified noise — hence the need to suppress small-\sigma terms.