The Bayesian Formulation

This is the fullest, most powerful way to pose an inverse problem — and, remarkably, it fits in one line. Bayes' theorem says the posterior distribution of the unknown model given the data is proportional to the likelihood (how well a model explains the data) times the prior (what we believed about the model beforehand). That is the entire idea. Everything else on this page is unpacking it.

p(m \mid d) \;\propto\; \underbrace{p(d \mid m)}_{\text{likelihood}}\; \underbrace{p(m)}_{\text{prior}}.

What makes this so much more than a formula is that three separate things you used to bolt on by hand are now all present at once: regularization, uncertainty, and prior knowledge. They are not add-ons; they are consequences. Where the statistical view told us the answer is a distribution, the Bayesian formulation tells us exactly how to build that distribution.

The posterior is not a single model

The Bayesian view reframes the whole enterprise. The answer to an inverse problem is not a single model but a probability distribution over models — the posterior p(m \mid d), which captures both the best estimate and how sure we are of it. The likelihood carries the physics and the noise; the prior carries everything we knew before the experiment (smoothness, positivity, a rough magnitude).

Read the two factors as two voices. The likelihood p(d\mid m) is the data speaking: "models that reproduce my measurements score high." A Gaussian noise model makes this voice a least-squares data-misfit term. The prior p(m) is prior expectation speaking: "models that are smooth / small / positive score high." A Gaussian prior makes this voice a regularization term. The posterior is the two voices multiplied — the models both voices like.

Gaussian everything

Take Gaussian noise e \sim N(0, C_D) and a Gaussian prior m \sim N(m_{\text{prior}}, C_M). Both factors are Gaussian, so the posterior is Gaussian too, and its peak — the MAP estimate — minimises the sum of two Mahalanobis terms:

\hat m = \arg\min_m\Big[(d - Gm)^{\mathsf T}C_D^{-1}(d - Gm) + (m - m_{\text{prior}})^{\mathsf T}C_M^{-1}(m - m_{\text{prior}})\Big], \hat m = m_{\text{prior}} + \big(G^{\mathsf T}C_D^{-1}G + C_M^{-1}\big)^{-1}G^{\mathsf T}C_D^{-1}\,(d - G m_{\text{prior}}).

The prior term C_M^{-1} is added to G^{\mathsf T}C_D^{-1}G — and that addition is exactly what makes the matrix invertible. The prior is the regularizer, a point the next page makes precise.

Worked example — Tikhonov falls out of the negative log

Let's watch the famous Tikhonov functional appear on its own. Suppose the noise is isotropic Gaussian, C_D = \sigma_D^2 I, and the prior is isotropic Gaussian centred at zero, C_M = \sigma_M^2 I. Maximising the posterior is the same as minimising its negative logarithm (the log turns the product into a sum, and Gaussians have e^{-(\cdot)} shapes). Taking -\log of p(d\mid m)\,p(m) gives

-\log p(m\mid d) \;=\; \frac{1}{2\sigma_D^2}\|Gm - d\|^2 \;+\; \frac{1}{2\sigma_M^2}\|m\|^2 \;+\; \text{const}.

Multiply through by 2\sigma_D^2 (which doesn't move the minimiser) and it reads

\hat m = \arg\min_m\Big[\|Gm - d\|^2 + \lambda^2\|m\|^2\Big], \qquad \lambda^2 = \frac{\sigma_D^2}{\sigma_M^2}.

That is the familiar Tikhonov functional, exactly — and it wasn't invented, it was derived. Even better, the mysterious regularization parameter \lambda now has a meaning: it is the ratio of the noise variance to the prior variance. Noisy data (large \sigma_D) or a confident prior (small \sigma_M) both push \lambda up, leaning harder on the prior. Clean data or a vague prior push it down, leaning harder on the measurement. The dial was never arbitrary.

Prior and data combine

The one-parameter picture: a prior belief about a model component, the likelihood the data provides, and the posterior that fuses them. Tighten the prior (small C_M) to lean on prior knowledge; tighten the data (small C_D) to lean on the measurement. Either way the posterior is narrower than both — combining information always reduces uncertainty.

Bayes is honest, but it is not magic. The posterior is only as trustworthy as the two ingredients you feed it: the prior and the noise model. Choose a smooth Gaussian prior when the true solution is actually sparse and edgy — a piecewise-constant image, a spiky spectrum — and the method will confidently smear those edges into gentle ramps. Misjudge the noise level \sigma_D and you set the wrong \lambda: too small and noise leaks through, too large and real features get flattened.

The saving grace is that the framework is honest about what it assumes — every assumption is written down as a distribution you can inspect and change. But it cannot rescue you from a wrong assumption. So the prior is a genuine modelling choice, on a par with choosing the physics, not a neutral technicality to be picked by default. Choosing it well is where the craft lives.

Step back and the Bayesian formulation is the grand unifying picture of inverse problems. It shows that regularization was never an arbitrary hack — it is a precise statement of prior belief, spelled out in the open. It shows that the real "answer" is a whole posterior distribution, from which you can extract not just a best estimate but full error bars (the posterior covariance).

And it shows that inverse problems are the same machinery humming inside a startling amount of modern technology. The Kalman filter steering a rocket or a robot vacuum is a recursive Bayesian update. The denoiser in your phone's camera is a posterior with an image prior. Medical scanners reconstruct pictures the same way, and much of modern AI is Bayesian inference wearing a large hat. Learn p(m\mid d) \propto p(d\mid m)\,p(m) once, and you have met the engine of half the century.