The Multivariate Gaussian

You already know the bell curve: one variable, a single hump centred on its average, thinning out symmetrically on each side. Now imagine a cloud of points scattered across a floor — heights and weights of a thousand people, say. Stand back and the cloud has a shape: densest in the middle, fading at the edges, and often stretched at an angle because taller people tend to be heavier. That fuzzy, tilted blob is a multivariate Gaussian — the bell curve grown into two, three, or a hundred dimensions.

The remarkable thing is how little it takes to describe the whole cloud. Two objects do it completely: a mean vector \boldsymbol\mu that says where the cloud is centred, and a covariance matrix \Sigma that says how it is shaped and tilted. Nothing else. Its density is

p(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^n \det\Sigma}}\,\exp\!\Big(-\tfrac12 (\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)\Big).

The exponent is the heart of it. The quadratic form (\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu) is the squared Mahalanobis distance — distance from the centre measured in units of standard deviation, stretched and rotated by the covariance. It reappears verbatim as the data-misfit term in the Bayesian inverse problem.

In 2-D it is a hill

Picture the two-dimensional case as a landscape: the density p(x, y) is the height of a smooth hill sitting above the plane. It is tallest right over the mean and slopes away in every direction. Slice that hill horizontally at some fixed height and the outline you trace is an ellipse. Stack those slices — one for each height — and you get a set of nested elliptical contours, exactly like the height lines on a map of a rounded hill.

The shape of those ellipses is the whole story, and it is set entirely by \Sigma:

Circular contours — the variables are independent with equal spread. The cloud is a round, featureless blob. Here \Sigma = \sigma^2 I.
Axis-aligned ellipses (a stretched oval, upright or flat) — independent, but one variable is more spread out than the other. Here \Sigma is diagonal with unequal entries.
Tilted ellipses — the variables are correlated. A positive off-diagonal covariance tilts the ellipse along the rising diagonal (big-x goes with big-y); a negative one tilts it the other way.

The ellipse's axes are the eigenvectors

There is a clean geometric dictionary between the matrix and the picture. The eigenvectors of \Sigma point along the axes of the ellipse, and the eigenvalues tell you how far the cloud spreads along each of those axes (the semi-axis lengths grow like \sqrt{\lambda}). A big eigenvalue is a long axis; a small one is a short, pinched axis.

So diagonalising the covariance matrix literally means finding the natural axes of the cloud — the directions along which the variables become uncorrelated. That is exactly what principal component analysis does: the first principal component is the longest axis of this very ellipsoid.

A special fact peculiar to Gaussians: zero covariance implies independence. For a general distribution, "uncorrelated" does not mean "independent" — but for jointly Gaussian variables a diagonal \Sigma means the components are genuinely, fully independent.

Reading the ellipse from the matrix

The nested ellipses below are equal-density contours of a centred 2-D Gaussian. Adjust \sigma_x, \sigma_y and the correlation \rho: the inner ellipse is the most probable region, and each outer ring is less likely. The whole family rotates and stretches together because they all share the one matrix \Sigma. Try to reproduce each of the three cases below in the figure.

Worked example 1 — circle or ellipse? Take

\Sigma = \begin{pmatrix} 4 & 0 \\ 0 & 4 \end{pmatrix}.

The diagonal entries are equal and the off-diagonal is zero, so the contours are perfect circles of radius set by \sqrt 4 = 2. The two variables are independent with equal spread — a round blob, no preferred direction. Change the bottom-right entry to 1 and the circle squashes into an axis-aligned ellipse, wide in x and short in y (still independent, just unequal spreads).

Worked example 2 — which way does it tilt? Now take

\Sigma = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix}.

The off-diagonal covariance is +2 — positive — so the ellipse tilts up along the diagonal: large x tends to go with large y. Its eigenvectors are (1,1) (eigenvalue 5, the long axis) and (1,-1) (eigenvalue 1, the short axis) — so the cloud is a cigar pointing along the 45^\circ line. Flip the covariance to -2 and the cigar swings to point along the falling diagonal instead.

Worked example 3 — moving the whole cloud. The mean vector does none of the shaping; it only relocates. Going from \boldsymbol\mu = (0,0) to \boldsymbol\mu = (5, -2) slides the entire family of ellipses rigidly five to the right and two down. Same shape, same tilt, same spread — a translation, nothing more. Shape lives in \Sigma; position lives in \boldsymbol\mu.

Those flat contours are slices through a hill. Here is the same 2-D Gaussian as a rotatable surface — the bell curve grown a dimension, a smooth mound of probability density tallest over the mean and fading outward. Drag it to see the round bell; the nested ellipses above are simply this surface's height lines viewed from directly overhead.

Fully described by mean vector \boldsymbol\mu (its centre) and covariance \Sigma (its shape and orientation).
The exponent is the Mahalanobis distance (\mathbf{x}-\boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu); level sets are ellipses whose axes are the eigenvectors of \Sigma.
For Gaussians, zero covariance ⇒ genuine independence.

The single most common blunder is to model each variable's variance separately and glue them together — in effect assuming \Sigma is diagonal when it isn't. That throws away the covariances, and the covariances are exactly where the correlation lives.

The consequences are not subtle. A truly correlated cloud is a tilted, narrow cigar; pretend it is diagonal and you draw a fat upright ellipse instead — a completely wrong shape. A point sitting just off the true cigar (very unlikely) can look perfectly ordinary under the fake circular model, so your probability estimates come out badly wrong. The correlations are not decoration: they are part of the distribution's identity. A multivariate Gaussian is defined by the covariance matrix, off-diagonals and all — never by treating the variances one at a time.

Look almost anywhere in quantitative science and the multivariate Gaussian is quietly holding things up:

It is the noise model behind the Kalman filter that tracks spacecraft, aircraft, and the GPS in your phone — the "cloud of uncertainty" around a position estimate is a Gaussian that gets shifted and reshaped at every tick.
It is the foundation of Gaussian processes in machine learning, where an entire function is treated as one enormous multivariate Gaussian.
It is the shape that PCA analyses — the ellipsoid whose axes are the principal components.
It is the distribution assumed in countless statistical methods, from linear regression to hypothesis tests.

Why so ubiquitous? Two deep reasons. First, it is the maximum-entropy distribution for a given mean and covariance — the least-assuming choice, the one that commits to nothing beyond those two facts. Second, the central limit theorem works in many dimensions too: add up lots of small independent influences and, whatever they were individually, the total drifts toward a multivariate Gaussian. Nature keeps handing us sums of many small things, so nature keeps handing us this bell-shaped cloud.