So far the model has been a fixed unknown and the noise an annoyance. The
statistical view takes the noise seriously: the data is a random
quantity, and so any estimate built from it is random too. We write
d = G m_{\text{true}} + e, \qquad e \sim N(0, C_D),
with e a random noise vector of
covariance
C_D. Repeat the experiment and you get different data, hence a different
estimate \hat m. The estimate is a random variable with its own
distribution — a spread of answers around the truth.
Bias, variance, and what a good estimator means
Treating \hat m as random lets us ask sharp questions. Is it
unbiased — is its average the true value, \mathbb{E}[\hat m] = m_{\text{true}}?
What is its variance — how widely does it scatter? The pseudoinverse estimate is
unbiased but, for ill-posed problems, has enormous variance (the noise blow-up).
Regularization deliberately accepts a little bias to cut that variance down — the
same bias–variance trade-off
seen in the resolution matrix, now stated probabilistically.
This reframing is the gateway to the Bayesian approach. Once data and noise are random, the tools
of likelihood and
Bayes apply directly,
and "solve the inverse problem" becomes "infer a distribution over models".
- Data is random: d = G m_{\text{true}} + e with e \sim N(0, C_D).
- Any estimate \hat m is therefore a random variable, with a bias and a variance.
- Regularization trades a little bias for a large reduction in variance — the statistical face of stability.