Priors Are Regularization

Up to now, regularization has looked like a slightly embarrassing trick: your least-squares fit blew up, so you added a penalty to calm it down, and you tuned a mysterious knob \alpha until the answer looked sensible. Where does that penalty come from? Why \|m\|^2 and not something else? And what on earth is \alpha?

The Bayesian view answers all three at once, and the answer is startlingly clean: Tikhonov regularization is exactly what you get when you assume the solution has a Gaussian prior probability distribution and the noise is Gaussian, and then ask for the most probable solution given the data. The penalty is not a trick bolted on afterwards — it is the negative logarithm of a prior belief. And the mysterious \alpha turns out to be a precise, interpretable ratio: how noisy you think the data is, divided by how large you expect the model to be.

Here is the punchline that unites the two halves of this course. Take the Bayesian MAP objective with a zero-mean Gaussian prior m \sim N(0, C_M) and uniform Gaussian noise C_D = \sigma^2 I. The negative log-posterior is, up to constants,

-\log p(m\mid d) \;=\; \frac{1}{\sigma^2}\|d - Gm\|^2 \;+\; m^{\mathsf T} C_M^{-1} m.

That is exactly a general Tikhonov functional. The data-fit term is the likelihood; the penalty term is the prior. Maximising the posterior is regularized least squares.

Where the equivalence comes from

Bayes' theorem says the posterior — what you believe about m after seeing the data — is proportional to the likelihood times the prior:

p(m \mid d) \;\propto\; p(d \mid m)\, p(m).

Now feed in Gaussians. Gaussian measurement noise means the likelihood is

p(d\mid m) \propto \exp\!\Big(-\tfrac{1}{2\sigma^2}\|d - Gm\|^2\Big),

and a zero-mean Gaussian prior on the model is

p(m) \propto \exp\!\Big(-\tfrac{1}{2\tau^2}\|m\|^2\Big).

Because these are exponentials of quadratics, taking -\log of the product turns the multiplication into a sum and strips the exponentials away, leaving a plain sum of squared norms. The MAP estimate — the model that maximises the posterior — is therefore the model that minimises

\frac{1}{2\sigma^2}\|d - Gm\|^2 + \frac{1}{2\tau^2}\|m\|^2,

which, after multiplying through by 2\sigma^2, is the Tikhonov functional \|d-Gm\|^2 + \alpha^2\|m\|^2 with \alpha^2 = \sigma^2/\tau^2. Maximising a Gaussian posterior and minimising a Tikhonov objective are the same act in two languages.

The dictionary

Term by term, the Bayesian and deterministic pictures are the same object in two languages:

So the regularization parameter was never arbitrary: \alpha is the ratio of how noisy you think the data is to how large you expect the model to be. And the penalty was never a mere mathematical trick — it is a precise statement of prior belief. The negative log-prior is the penalty, and the parabola m^2/(2\tau^2) is the L2 penalty drawn out.

Worked examples: reading the knobs

1. The meaning of \alpha

Because \alpha^2 = \sigma^2_{\text{noise}}/\sigma^2_{\text{prior}}, the regularization strength is a tug-of-war between two variances. Suppose your sensors are noisy, \sigma^2_{\text{noise}} = 4, and you expect a fairly large model, \sigma^2_{\text{prior}} = 100. Then \alpha^2 = 4/100 = 0.04 — light regularization, because you mostly trust that the model can be large. Now quieten the sensors to \sigma^2_{\text{noise}} = 0.01: \alpha^2 = 0.0001, almost no regularization, because clean data should be trusted. More noise ⇒ more regularization; a tighter prior ⇒ more regularization. Both are exactly what intuition demands, and now they fall out of a formula rather than a hunch.

2. A tighter prior pulls harder

Hold the noise fixed at \sigma^2_{\text{noise}} = 1. If you strongly believe the model is small, set a tight prior \sigma^2_{\text{prior}} = 1, giving \alpha^2 = 1. Loosen your belief to \sigma^2_{\text{prior}} = 25 and \alpha^2 = 0.04 — the penalty weakens twenty-fivefold and the fit is allowed to roam. The width of your prior is the strength of your penalty, inverted.

3. A smoothness penalty is a correlated prior

The general penalty \|Lm\|^2 from smoothing Tikhonov corresponds to a prior with covariance C_M = (L^{\mathsf T}L)^{-1}. A diagonal L = I means the model components are independent under the prior — knowing one tells you nothing about its neighbour. But a derivative L = D_2 makes C_M couple neighbouring values: the prior now says "adjacent points are correlated, so they should look alike." That correlation is exactly what makes the reconstruction smooth. A smoothness penalty and a smoothness prior are the same statement.

The penalty drawn from the prior

The bell curve is a zero-mean Gaussian prior on a model component; the upward parabola is its negative logarithm — the penalty m^2/(2\tau^2) that regularization adds. Narrow the prior (smaller \tau, more confident) and the parabola steepens — a stronger pull toward zero, i.e. a larger \alpha.

Digging deeper

The Bayesian reading is illuminating, but it comes with a warning printed in the fine text. Your regularized solution is only ever as good as your prior assumptions. The MAP estimate is the most probable model under the prior you chose — and if the true solution is nothing like a sample from your assumed Gaussian, that "most probable" answer can be confidently, systematically wrong.

Suppose the truth is sparse — mostly zeros with a few big spikes — or has hard edges, neither of which looks anything like a smooth Gaussian draw. A Gaussian prior will spread each spike into a gentle bump and blur each edge into a ramp, and it will do so with total statistical confidence. The optimisation succeeded perfectly; it just optimised the wrong objective. No amount of clean data or careful tuning of \alpha can rescue a bad modelling assumption. The math cannot fix a wrong prior — it can only faithfully carry it through to a wrong answer.

It really is. Once you see regularization as "likelihood times Gaussian prior", an astonishing range of tools collapse into one family, all running the same Bayesian machinery with Gaussian priors under the hood:

And here is the jump that changes everything: swap the Gaussian prior for a sparsity prior — a Laplace distribution, whose negative log is |m| rather than m^2 — and the L2 penalty becomes the L1 / LASSO penalty behind compressed sensing. That is the technology that lets an MRI scanner reconstruct a sharp image from a fraction of the usual measurements, imaging you in a fraction of the usual time. Different prior, different penalty, same beautiful machine.

The whole equivalence rides on one small algebraic miracle. A Gaussian is e^{-(\text{quadratic})}. Multiplying the likelihood by the prior multiplies two such exponentials; taking -\log converts that product into a sum of the quadratics and deletes the exponential entirely. So "maximise a product of Gaussians" becomes "minimise a sum of squared norms" — and a sum of squared norms is precisely a least-squares-plus-penalty objective. Gaussians are the one prior family for which the negative log is a clean quadratic, which is exactly why they, and not some other bell-ish curve, give you Tikhonov on the nose.