Imagine a spreadsheet with a thousand columns — every gene measured in a cell, every pixel of a face, every sensor on an engine. A thousand numbers per sample is far too many to plot, to reason about, or to feed cheaply into a model. Yet almost always those thousand numbers are not really independent: they move together, they echo each other, they trace out a much thinner shape hiding inside the vast space. Principal component analysis (PCA) is the tool that finds that hidden shape — the handful of directions along which the data actually varies — and lets you throw the rest away with almost no loss.

It is the workhorse of dimensionality reduction, and — wonderfully — it is nothing but eigenvectors of a symmetric matrix in disguise. Every idea you need is already built: the covariance matrix that records how features vary together, and the spectral theorem that guarantees its eigenvectors are perpendicular. PCA simply reads those eigenvectors off, biggest first, and calls them the principal components.

The recipe, in four steps

Lay your data in a matrix X with one row per sample and one column per feature. PCA is then a short, fixed routine:

• Centre it. Subtract each column's mean so the cloud sits at the origin. This step is not optional — skip it and everything below breaks (see the "Watch out!" box).
• Form the covariance matrix C = \tfrac{1}{n-1} X^{\mathsf T} X (using the centred X). It is d \times d for d features, always symmetric, and positive semidefinite — no direction can have negative variance.
• Diagonalize it. Find the eigenvalues \lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_d \ge 0 and their perpendicular eigenvectors \vec v_1, \vec v_2, \dots. These eigenvectors are the principal components; each eigenvalue is the variance captured along its direction.
• Keep the top k. Project every sample onto \vec v_1, \dots, \vec v_k. That gives the best possible k-dimensional summary of the data — the one that loses the least, in the exact sense of minimising reconstruction error.

The first eigenvector \vec v_1 points along the direction of maximum variance — the long axis of the cloud. The second is the perpendicular direction of next-most variance, and so on down the list. Because the matrix is symmetric, those directions are automatically at right angles: a clean, ranked, orthogonal set of axes fitted to the data itself.

Worked example 1: the long axis of a tilted cloud

Take a small centred dataset of five points already sitting at the origin: (-2,-1),\ (-1,-1),\ (0,0),\ (1,1),\ (2,1). They lean along a diagonal — as one coordinate grows the other tends to grow too. Build the 2\times2 covariance matrix with C = \tfrac{1}{n-1}X^{\mathsf T}X, so we divide by n-1 = 4:

Now find its top eigenvector — the first principal component. The characteristic equation is (2.5-\lambda)(1-\lambda) - 1.5^2 = \lambda^2 - 3.5\lambda + 0.25 = 0, giving \lambda = \tfrac{3.5 \pm \sqrt{12.25 - 1}}{2}, i.e. \lambda_1 \approx 3.427 and \lambda_2 \approx 0.073. For the larger eigenvalue, solving (C - \lambda_1 I)\vec v = 0 gives the direction \vec v_1 \approx (1,\ 0.618) — which points, exactly as promised, straight along the long diagonal axis of the cloud. That single direction is what PCA would keep if you asked it to squeeze this 2-D data down to one dimension.

Worked example 2: explained variance

How much did we actually keep? Each eigenvalue is a variance, so the fraction of the total spread captured by component i — the explained variance ratio — is simply

Suppose a dataset's covariance matrix has just two eigenvalues, 9 and 1. Then the total variance is 9 + 1 = 10, and the first principal component alone captures

Keeping only that one direction throws away a mere 10\% of the variance while halving the number of dimensions. That is the whole promise of PCA in one line: when a few eigenvalues dwarf the rest, a few directions carry nearly all the information. (For the tilted cloud of example 1, the first component captures 3.427 / (3.427 + 0.073) \approx 98\% — the second direction is almost pure thinness.)

See it: the data ellipse and its principal axes

Here is a centred cloud of correlated points. The two arrows are the principal components — the eigenvectors of the cloud's covariance matrix — each drawn along its own direction and scaled by \sqrt{\lambda}, the standard deviation of the data along that axis. Together they trace the data ellipse. Turn up the correlation and watch the long axis swing to follow the tilt; stretch each spread and watch the arrows grow. The readout shows the two eigenvalues and how much variance the first component explains — crank the correlation and the first component swallows nearly everything.

PCA is just the SVD in disguise

You do not actually have to build C = \tfrac{1}{n-1}X^{\mathsf T}X and diagonalize it. Run the singular value decomposition straight on the centred data matrix, X = U\Sigma V^{\mathsf T}, and the principal components fall out for free:

• The right singular vectors (columns of V) are exactly the principal components — the eigenvectors of X^{\mathsf T}X.
• The squared singular values are proportional to the eigenvalues: \lambda_i = \sigma_i^2 / (n-1), so the largest singular value marks the direction of greatest variance.

This is not just an algebraic curiosity — it is how PCA is computed in practice. Forming X^{\mathsf T}X squares the condition number and loses numerical precision; the SVD works directly on X and is far more stable. Keeping the top k singular directions is precisely the best rank-k approximation guaranteed by Eckart–Young. PCA, low-rank approximation, and the SVD are three views of one idea: keep the strong directions, drop the weak ones.