Diagonalization is a beautiful trick — but it demands a square matrix with a full set of independent eigenvectors, and plenty of matrices simply don't qualify. A photograph stored as pixel brightnesses is rectangular. A table of movie ratings (rows of users, columns of films) is rectangular. Most real-world data matrices are rectangular, noisy, and messy — nothing like the tidy square examples used to introduce eigenvectors.

The singular value decomposition (SVD) is the universal fix. It works on every matrix — square or rectangular, invertible or singular, symmetric or not — no exceptions, no special cases. It says any linear map, however complicated, is really just three simple moves chained together:

First a rotation (V^{\mathsf{T}}), then a pure axis-aligned stretch by the singular values in \Sigma, then another rotation (U). It's not an exaggeration to call the SVD the single most useful factorization in all of applied mathematics — it underpins compression, search, noise removal, and machine learning alike, all from one rotate–stretch–rotate idea.

The geometric picture

Read A = U\Sigma V^{\mathsf{T}} as a pipeline acting on an input vector, right to left:

• V^{\mathsf{T}} rotates the input space so that a special set of perpendicular directions — the right singular vectors, the columns of V — line up with the ordinary coordinate axes.
• \Sigma stretches along those aligned axes by the singular values \sigma_1 \ge \sigma_2 \ge \cdots \ge 0 — always non-negative, and always listed largest first.
• U rotates the stretched result into its final orientation in the output space, using another perpendicular set of directions — the left singular vectors.

For a general matrix, U and V are genuinely different rotations — the input and output spaces don't have to line up at all, and for a rectangular matrix they don't even have the same dimension. That flexibility is exactly what lets the SVD handle matrices that plain diagonalization can't touch.

Here's the trick that makes the SVD buildable out of tools you already have: even when A itself is rectangular or defective, the product A^{\mathsf{T}}A is always square and always symmetric, which guarantees it a full, perpendicular set of eigenvectors with non-negative eigenvalues. Those eigenvectors of A^{\mathsf{T}}A become the columns of V, and the square roots of its eigenvalues become the singular values \sigma_i. The SVD of any matrix, however messy, is assembled entirely from the diagonalization of one friendly symmetric matrix built from it. Symmetrically, the columns of U come from AA^{\mathsf{T}} — a different symmetric matrix (a different size, too, whenever A is rectangular) that shares the very same non-zero eigenvalues, which is exactly why the two rotations sandwich a single shared list of singular values in the middle.

Worked example: a circle becomes an ellipse

Here's the picture that says it all. Take A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} — the same matrix from diagonalization, chosen here because it's symmetric and positive-definite, which makes its SVD easy to see by eye. Any matrix turns the unit circle into an ellipse, and the lengths of that ellipse's two perpendicular semi-axes are exactly the singular values. Apply the transformation below and watch the circle swell into its ellipse, principal axes marked.

Because this particular A is symmetric and positive-definite, something neat happens: its singular values (3 and 1) turn out to be exactly the same as its eigenvalues, and U and V coincide — the SVD and the eigendecomposition collapse into the very same rotate-then-stretch picture. That's a special coincidence of symmetric positive-definite matrices, not a general rule (more on that in the "Watch out!" box below).

Worked example: singular values scale area, just like the determinant

Stretching the unit circle into an ellipse with semi-axes \sigma_1 and \sigma_2 multiplies its area by \sigma_1\sigma_2 — rotations never change area, only the stretch does. That means the singular values quietly reproduce the determinant's "area scaling factor" idea:

Check it against the worked matrix above: \det A = 2\cdot 2 - 1\cdot 1 = 3, and indeed \sigma_1\sigma_2 = 3 \times 1 = 3. It matches exactly. (For a general, non-symmetric matrix the eigenvalues can be negative or even complex and don't multiply out to |\det A| so cleanly by themselves — but the singular values, being genuine stretch factors, always do.)

Worked example: throwing away the smallest singular value

Now the payoff. Every matrix can be rebuilt as a sum of simple rank-1 pieces, one per singular value, largest first: A = \sigma_1\vec{u}_1\vec{v}_1^{\mathsf{T}} + \sigma_2\vec{u}_2\vec{v}_2^{\mathsf{T}} + \cdots. Stop after the first term and you get the best possible rank-1 approximation of A. For our matrix, \sigma_1 = 3 along the direction (1,1)/\sqrt{2}, so the rank-1 approximation is

compared with the true A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}. Every entry is within 0.5 of the original, using a matrix built from just one number (\sigma_1=3) and one direction, instead of all four original entries. Keep both singular values and you rebuild A exactly — the small remaining singular value \sigma_2=1 is exactly what's missing.

Why the SVD rules data science

The singular values rank the directions by how much the matrix stretches them — so the largest few capture most of what the matrix does, and the tiny ones can often be thrown away entirely. That is exactly how you compress an image, denoise data, build recommender systems, and run principal component analysis — keep the strong directions, drop the weak ones. It's the perfect capstone for everything built in this course: rotations, stretches, eigenvectors, and orthogonality, all folded into one decomposition that works on absolutely any matrix you hand it.