SVD and the Pseudoinverse
The
singular value decomposition
is the master tool of inverse theory. Every forward operator factors as
G = U\Sigma V^{\mathsf T} = \sum_{i} \sigma_i\, u_i v_i^{\mathsf T},
with orthonormal model directions v_i, orthonormal data directions
u_i, and singular values
\sigma_1 \ge \sigma_2 \ge \dots \ge 0. The map takes each model
direction v_i, stretches it by \sigma_i, and
lands it on data direction u_i.
The pseudoinverse just divides by σ
To invert, undo each step — divide by the singular value instead of multiplying:
G^{+} = V\Sigma^{+}U^{\mathsf T} = \sum_{\sigma_i > 0} \frac{1}{\sigma_i}\, v_i u_i^{\mathsf T}, \qquad \hat m = G^{+}d = \sum_{\sigma_i>0}\frac{u_i^{\mathsf T}d}{\sigma_i}\,v_i.
This one formula explains everything ahead. Directions with zero singular values
are dropped — they are the model null space, invisible to the data (the source of
non-uniqueness). Directions with tiny singular values get divided by a tiny
number, so the term (u_i^{\mathsf T}d)/\sigma_i explodes — any noise in
d along u_i is amplified by
1/\sigma_i. That is the engine of instability, and regularization is
precisely the art of taming those 1/\sigma_i factors.
When a singular value shrinks
The forward map sends the unit circle to an ellipse with semi-axes
\sigma_1, \sigma_2. Shrink \sigma_2 toward
zero and the ellipse collapses to a sliver — that model direction barely registers in the data.
Watch the readout: as \sigma_2 \to 0, the inverse factor
1/\sigma_2 blows up. Recovering that direction means dividing the data
(noise and all) by something tiny.
- G = U\Sigma V^{\mathsf T}; G^{+} = V\Sigma^{+}U^{\mathsf T} with \sigma_i \mapsto 1/\sigma_i (and 0 \mapsto 0).
- Zero singular values → model null space → non-uniqueness.
- Tiny singular values → factor 1/\sigma_i amplifies noise → instability.