Quantifying Uncertainty

Imagine two teams reconstruct the same underground structure from the same seismic data, and both report the identical best-fit image. Are the two answers equally trustworthy? Not necessarily — and the image alone cannot tell you. One team's answer might be pinned down tightly by the data; the other's might be a near-guess propped up by regularization. The difference — the error bars — is the whole point of a Bayesian treatment.

The Bayesian formulation hands you a full posterior distribution over the solution. Its mean is your best estimate — but its posterior covariance is the real prize: it tells you the uncertainty on every recovered parameter, and how those uncertainties are correlated. With Gaussian likelihood and prior the posterior is itself Gaussian, so it is fully described by a mean (the MAP / regularized solution) and the posterior covariance

C_{\text{post}} = \big(G^{\mathsf T} C_D^{-1} G + C_M^{-1}\big)^{-1}.

This single matrix is the uncertainty of the answer. Its diagonal entries are the variances of each recovered parameter — square-root them for error bars. Its off-diagonals say which parameters are correlated, i.e. which trade off against each other in the fit.

Reading a covariance matrix

A covariance matrix is a table you can read at a glance once you know the code. For a two-parameter model (m_1, m_2),

C_{\text{post}} = \begin{pmatrix} \sigma_1^2 & \rho\,\sigma_1\sigma_2 \\ \rho\,\sigma_1\sigma_2 & \sigma_2^2 \end{pmatrix}.

Diagonal \sigma_1^2, \sigma_2^2: the variance of each parameter. A large diagonal entry means that parameter is poorly determined; a small one means it is well determined. Error bar on m_1 is \sqrt{\sigma_1^2}.
Off-diagonal \rho\,\sigma_1\sigma_2: how the two errors are linked. \rho \approx 0 means the data pins down each parameter separately; \rho \approx \pm 1 means the data cannot separate them — only their combination is constrained.

Worked example: three little matrices

Read off what each of these posterior covariances is telling you.

C_A = \begin{pmatrix} 0.01 & 0 \\ 0 & 0.01 \end{pmatrix}, \quad C_B = \begin{pmatrix} 25 & 0 \\ 0 & 0.01 \end{pmatrix}, \quad C_C = \begin{pmatrix} 4 & 3.9 \\ 3.9 & 4 \end{pmatrix}.

C_A: both variances tiny (error bars \pm 0.1), no correlation. Both parameters well determined and independent — a clean, informative experiment.
C_B: m_1 has variance 25 (error bar \pm 5) while m_2 has \pm 0.1. The data nails m_2 but barely constrains m_1 — an ill-conditioned, poorly-resolved direction.
C_C: correlation \rho = 3.9/4 \approx 0.98, nearly perfect. The data cannot separate m_1 and m_2; it only pins down their combination (roughly m_1 - m_2 is tight, while m_1 + m_2 is loose). The uncertainty ellipse is a long thin diagonal cigar.

Large posterior variance is the signature of the ill-conditioned, poorly-resolved directions: where the data is uninformative, the posterior simply falls back to the prior's spread. The covariance and the resolution are two views of the same thing.

Resolution and uncertainty are two sides of one ellipse

The posterior is a multivariate Gaussian, and C_{\text{post}} is its uncertainty ellipse. Well-constrained directions (where the data is informative — large singular values) come out with small variance: the ellipse is thin there. Poorly constrained directions stay wide, inheriting the prior's spread. So the shape of the ellipse reports the resolution direction by direction.

Two ways to shrink it: gather more data (each measurement adds to the precision G^{\mathsf T}C_D^{-1}G) or impose a tighter prior (larger C_M^{-1}) — though a tighter prior shrinks uncertainty by adding bias, the same trade-off once more.

Watch the uncertainty shrink

The ellipse is the posterior covariance for a two-parameter model. With few measurements it is large and tilted (high, correlated uncertainty). Add measurements with the slider: each one adds precision, and the ellipse contracts — fastest along the well-measured direction, slowest along the poorly-measured one, so it does not just shrink but changes shape.

C_{\text{post}} = (G^{\mathsf T}C_D^{-1}G + C_M^{-1})^{-1} — the full uncertainty of the estimate.
Diagonal → variances → error bars; off-diagonal → parameter correlations.
More data or a tighter prior shrinks the ellipse; its shape reports direction-by-direction resolution.

A reconstruction's mean alone is dangerously incomplete without its posterior covariance. Two reconstructions can look pixel-for-pixel identical yet have wildly different reliability — one tightly constrained by data, the other a soft guess held up by regularization. Reporting an estimate with no covariance is like reporting a measurement with no error bars: it looks authoritative and tells you nothing about whether to believe it.

The posterior covariance is exactly what separates "the data really determined this" from "the prior guessed this." A parameter with huge posterior variance was not measured — it was assumed. Always ship the uncertainty with the answer.

A second trap: do not confuse a small posterior variance with a correct answer. Cranking up the prior's tightness shrinks the covariance beautifully, but it does so by injecting bias — the ellipse is small because you told it to be, not because the data earned it.

The posterior covariance is not a niche curiosity — it is the quantity a Kalman filter propagates and updates at every single step. A self-driving car, a spacecraft, or a phone's GPS does not merely estimate where it is; it carries a covariance matrix saying how sure it is — an uncertainty ellipse around its position.

Watch that ellipse breathe: every time a good measurement arrives (a crisp GPS fix, a landmark the camera recognises), the filter multiplies in a sharp likelihood and the ellipse shrinks. Between measurements, as the vehicle rolls forward on noisy wheels and drifts, the prediction step grows the ellipse again — uncertainty accumulates until the next fix reins it in. That constant shrink-and-grow of the covariance is how an autonomous system stays honest about its own ignorance.

This is the deep lesson of the whole subject: a best guess with no uncertainty is fragile, because you cannot tell a lucky guess from a firm measurement. Quantified uncertainty — the posterior covariance — is what turns a brittle estimate into something you can safely steer a car by.