MAP Estimation

You are handed a coin, flip it three times, and get heads, heads, heads. What is the coin's bias — its probability of landing heads? Maximum likelihood answers with cold arithmetic: three heads out of three tosses, so the best-fitting bias is 3/3 = 1.0 — a two-headed coin, guaranteed. But you have seen coins before. Almost all of them are close to fair. Three flips is flimsy evidence, and your gut says the truth is nearer 0.5 than 1.0.

That gut feeling has a name and a formula. Maximum a posteriori (MAP) estimation folds your prior belief together with the likelihood of the data, and picks the parameter that is most probable given both. It maximises the posterior:

\hat\theta_{\text{MAP}} = \arg\max_\theta\, p(\theta \mid D) = \arg\max_\theta\, \underbrace{p(D\mid\theta)}_{\text{likelihood}}\,\underbrace{p(\theta)}_{\text{prior}}.

Taking logs turns the product into a sum: MAP maximises \log p(D\mid\theta) + \log p(\theta). The first term is the data fit (the MLE objective); the second is a penalty that pulls the answer toward what the prior considers plausible. In one slogan: MAP = MLE + a prior nudge.

The three-heads coin, worked out

Let the bias be \theta. A natural prior for a probability is a Beta distribution \text{Beta}(\alpha,\beta); with heads count h out of n flips the posterior is \text{Beta}(\alpha+h,\ \beta+n-h), whose peak (the MAP estimate) is

\hat\theta_{\text{MAP}} = \frac{\alpha + h - 1}{\alpha + \beta + n - 2}.

A prior of \text{Beta}(2,2) gently says "coins are probably fair-ish". Feed in our data h=3,\ n=3:

\hat\theta_{\text{MAP}} = \frac{2 + 3 - 1}{2 + 2 + 3 - 2} = \frac{4}{5} = 0.8.

Far more sensible than the MLE's reckless 1.0: the three heads pull our estimate up, but the prior refuses to let it hit certainty on such thin evidence. A stronger prior \text{Beta}(10,10) ("I'm quite sure coins are fair") is even more stubborn — (10+3-1)/(10+10+3-2)=12/21\approx 0.57, barely budging from 0.5.

More data washes the prior out

The prior's grip is not permanent — it fades as evidence piles up. Keep the mild \text{Beta}(2,2) prior but now flip the coin thirty times and get thirty heads:

\hat\theta_{\text{MAP}} = \frac{2 + 30 - 1}{2 + 2 + 30 - 2} = \frac{31}{32} \approx 0.97.

With a mountain of data the likelihood shouts and the prior whispers, so MAP slides right up to the MLE's answer. This is the general rule: with lots of data MAP ≈ MLE; with little data the prior does the heavy lifting. A good prior is a lifeline when evidence is scarce and a harmless bystander once evidence is plentiful.

Gaussian prior, Gaussian likelihood

The same story plays out for a continuous quantity. With a Gaussian prior N(\mu_0, \tau^2) and Gaussian data of spread \sigma_d, the posterior is again Gaussian, and its peak — the MAP estimate — is a precision-weighted average of prior and data:

\hat\theta_{\text{MAP}} = \frac{\mu_0/\tau^2 + \bar x/\sigma_d^2}{1/\tau^2 + 1/\sigma_d^2}.

Confident prior (small \tau) pulls toward \mu_0; precise data (small \sigma_d) pulls toward \bar x. Two limits matter: a flat (uninformative) prior recovers the MLE, while in the Gaussian case the log-prior penalty is exactly \|\theta-\mu_0\|^2/\tau^2 — a squared penalty. That is the seed of ridge / Tikhonov regularization: a Gaussian prior is an L2 penalty.

Prior meets data

The three bumps are the prior (what we believed), the likelihood (what the data says), and their product the posterior — each drawn at unit peak so the shapes are easy to compare. The posterior sits between prior and data and is narrower than either (combining information always sharpens). Tighten the prior or the data and watch the posterior slide toward whichever you trust more — the peak of that green curve is the MAP estimate.

MAP maximises \log p(D\mid\theta) + \log p(\theta) — data fit plus a prior penalty.
Gaussian prior + Gaussian likelihood ⇒ Gaussian posterior; the MAP is a precision-weighted average.
A flat prior recovers the MLE; a Gaussian prior is an L2 (ridge / Tikhonov) penalty.

MAP hands you a single "best" number — the summit of the posterior. That is seductive, but a summit throws away the view. The full posterior distribution carries all of your uncertainty: how wide it is, whether it is lopsided, whether it has more than one hump. A MAP point estimate keeps none of that.

When the posterior is skewed, its peak can sit far from where most of the probability actually lives — reporting the mode alone quietly misleads. When it is multi-peaked, MAP crowns one hill and pretends the others do not exist, even if a rival hill is nearly as tall. A full Bayesian analysis keeps the entire posterior — mean, spread, credible intervals, every mode — not just its highest point. Use MAP for a quick answer; reach for the whole posterior when the shape matters.

Here is the punchline that ties statistics to modern machine learning with a single knot. Training a model by MAP means maximising \log p(D\mid\theta) + \log p(\theta) — and that prior term is exactly a regularizer:

a Gaussian prior on the parameters ⇒ an L2 / ridge penalty (the famous "weight decay");
a Laplace (double-exponential) prior ⇒ an L1 / LASSO penalty, the one that drives coefficients to exactly zero.

So every time a neural network uses weight decay, it is doing MAP estimation with a built-in prior belief that the weights ought to be small. The engineer who dialled up the regularization strength was, without saying so, expressing a religious conviction about the parameters — a narrower prior. The same knot appears in Tikhonov-regularized inverse problems: regularization and Bayesian priors are two names for one idea.