Principal Component Analysis

Dimensionality reduction made a promise: squeeze a pile of features down to just a few, and keep most of the data's real structure. But it left one question dangling — a smaller set of what, exactly? Which new axes should you actually draw, and how do you find the best ones without endless trial and error?

Principal component analysis (PCA) answers with a precise recipe. It finds the directions of greatest variance in the data — the principal components — and keeps only the strongest few. The first principal component is the single line along which the data is most spread out; the second is the best direction perpendicular to it, and so on.

Here is the payoff of the whole journey. Those principal directions are exactly the eigenvectors of the data's covariance matrix — which, being symmetric, is guaranteed to hand us perpendicular axes. The eigenvalue of each is how much variance that direction carries. All that abstract eigenvector machinery from linear algebra turns out to be exactly the right tool for this one very concrete job.

The recipe, step by step

Turning "find better axes" into an actual algorithm takes four steps:

  1. Center the data. Subtract each feature's mean from every value of that feature, so the whole cloud of points balances around the origin. If your features sit on very different scales — kilograms next to kilometres — scale them first too, or the feature with the biggest raw numbers will dominate the variance for no good reason.
  2. Build the covariance matrix. This one matrix records, for every pair of features, how much they vary together — the covariance matrix is the whole shape of the data cloud, squeezed into a single symmetric table of numbers.
  3. Find its eigenvectors. The eigenvectors of the covariance matrix are the principal components — the directions of greatest spread — and their eigenvalues rank them by exactly how much variance each one captures.
  4. Project onto the top few. Drop every axis except the handful with the largest eigenvalues, and project each data point onto just those — a lower-dimensional stand-in for the original data that keeps as much real structure as any same-sized stand-in possibly could.

In practice, most software never even builds the covariance matrix explicitly — it reads the principal components straight off the singular value decomposition of the centered data, which is faster and more numerically stable. Both routes land on exactly the same principal components; the covariance-matrix version is just easier to picture first.

Find the principal direction

Rotate the candidate axis and watch the variance of the projected points. It peaks when the axis lines up with the long stretch of the cloud — that maximum-variance direction is the first principal component, PC_1. The perpendicular one is PC_2. Keeping only PC_1 compresses the data to one dimension while preserving the most information possible.

Notice what just happened with a real, worked cloud of points: the long axis of this stretched, egg-shaped scatter is PC_1 — nobody had to guess it, the variance calculation found it automatically. The short axis across the narrow width of the egg is PC_2, carrying only the little bit of spread left over. Because almost all the spread lives along the long axis, throwing PC_2 away and keeping only each point's position along PC_1 turns this 2-D cloud into a 1-D list of numbers with barely any real loss — the same diagonal-line story dimensionality reduction started with, now solved exactly rather than by eye.

Why it matters: clouds live in many dimensions

Real data rarely sits in a tidy 2-D plane — but the same idea scales. Here is a cloud of points in three dimensions; drag to rotate it. Notice it isn't a shapeless blob: it's a flattened, tilted pancake. The bold line is PC_1, the single direction along which the points spread the most. Projecting the whole cloud onto that one axis (and maybe the next) throws away the thin dimensions and keeps the shape that matters — that is PCA turning three numbers per point into one or two, with almost no loss.

A tiny worked example, start to finish

Four data points, two features each — small enough to run the whole recipe by hand: (5,4), (-1,-2), (3,0), (1,2). Plotted, they already look like a short, stretched ellipse leaning along a diagonal, with two points lying nearly on that diagonal and two sitting off to either side of it. Real datasets run this same four-step recipe over thousands of points and dozens or hundreds of features at once — a computer just automates exactly this hand-arithmetic, at scale.

Step 1 — center. The mean is (2, 1) (average the x's, then the y's). Subtracting it from every point gives the centered cloud (3,3), (-3,-3), (1,-1), (-1,1) — balanced around the origin, exactly as the recipe demands.

Step 2 — covariance. Average the squares and cross-products of those centered coordinates:

\operatorname{cov}(x,x) = \tfrac{3^2+(-3)^2+1^2+(-1)^2}{4} = 5, \qquad \operatorname{cov}(y,y) = \tfrac{3^2+(-3)^2+(-1)^2+1^2}{4} = 5 \operatorname{cov}(x,y) = \tfrac{(3)(3)+(-3)(-3)+(1)(-1)+(-1)(1)}{4} = 4

That assembles into one symmetric covariance matrix, ready for steps 3 and 4 next:

C = \begin{pmatrix} 5 & 4 \\ 4 & 5 \end{pmatrix}

How many components do you actually need? Read the eigenvalues

An eigenvalue isn't just a ranking number — it is the exact amount of variance its principal component carries. Add up every eigenvalue and you get the dataset's total variance; divide any one eigenvalue by that total and you get the precise fraction of the data's spread you keep (or throw away) by dropping that component. No guessing required.

Take that covariance matrix from the tiny four-point example above:

C = \begin{pmatrix} 5 & 4 \\ 4 & 5 \end{pmatrix}

Its eigenvalues solve (5-\lambda)^2 - 4^2 = 0, so 5 - \lambda = \pm 4, giving \lambda_1 = 9 and \lambda_2 = 1. Their eigenvectors point along the two diagonals, (1,1)/\sqrt{2} and (1,-1)/\sqrt{2} — so PC_1 runs exactly along the direction where the two features rise and fall together, which is precisely the long diagonal the four points leaned along in the first place.

Total variance is 9 + 1 = 10, so PC_1 alone accounts for 9/10 = 90\% of it. Keep just that one number per data point instead of two, and you've thrown away only a tenth of the information — often stated as "the first component captures 90% of the variance." Real datasets with lots of redundant features can do even better, sometimes over 99% with a single component. Check the arithmetic yourself:

// A 2x2 covariance matrix for two positively-correlated features. const C: number[][] = [ [5, 4], [4, 5], ]; // For a symmetric 2x2 matrix [[a, b], [b, d]], the eigenvalues solve // (a - lambda)(d - lambda) - b*b = 0. function eigenvalues2x2(a: number, b: number, d: number): [number, number] { const trace = a + d; const det = a * d - b * b; const disc = Math.sqrt(trace * trace - 4 * det); return [(trace + disc) / 2, (trace - disc) / 2]; } const [lambda1, lambda2] = eigenvalues2x2(C[0][0], C[0][1], C[1][1]); const total = lambda1 + lambda2; console.log(`PC1 eigenvalue: ${lambda1}`); console.log(`PC2 eigenvalue: ${lambda2}`); console.log(`Variance kept by PC1 alone: ${((lambda1 / total) * 100).toFixed(1)}%`);

Try changing the numbers in C and rerunning. Push the off-diagonal term up towards 5 and PC₁'s share climbs towards 100\% — the two features are becoming almost perfectly correlated, so one direction carries nearly everything. Set it to 0 instead and the two eigenvalues split evenly, 50\% each: the features were already uncorrelated, the covariance matrix was already diagonal, and PCA has nothing useful to rotate — the original axes were the principal components all along.

Two classic PCA traps catch almost everybody:

Treat every pixel of a grayscale photograph as one feature, and a single face photo becomes a vector with tens of thousands of entries. In 1991, researchers Matthew Turk and Alex Pentland ran PCA on a dataset of face photos built exactly this way — and the resulting principal components, reshaped back into images, turned out to look like blurry, ghostly, half-formed faces. They named them eigenfaces.

Every real face in the dataset could be rebuilt, almost exactly, as a blend of just a few dozen eigenfaces instead of tens of thousands of raw pixel values — an early, genuinely practical face-recognition system built entirely on the eigenvectors of a covariance matrix. It's the payoff this whole linear-algebra unit was quietly building towards: an eigenvector isn't just an abstract arrow on a page any more, it's a ghostly little portrait doing real work.

Why this is the perfect ending

PCA ties the whole Primer together. It needs dot products to project, a symmetric matrix to summarise the spread, and its eigenvectors to find the best axes — every idea earning its keep at once. It powers data visualisation (squash a hundred features down to two or three you can actually plot), noise reduction (the low-variance components are often mostly noise, so dropping them cleans the signal up), face recognition, and general-purpose compression. From an arrow on a page to the principal components of real data, it's been one connected story: linear algebra is the language, and machine learning is what it lets you say.

See it explained