Truncated SVD

Once you know that the trouble in an ill-posed problem comes entirely from the tiny singular values — the directions the measurement barely saw, which explode when you divide by them — the simplest fix in the world suggests itself: just throw them away.

Keep the large singular values, which correspond to the strong, reliable, well-measured directions of your data. Discard the small ones, dominated by noise, that would otherwise blow up in your face. It is brute-force and unsubtle, and it is also the most intuitive regularization method there is — you can see exactly why it works. This is the truncated SVD.

The bluntest regularizer simply throws away the dangerous directions. Since the trouble lives in the terms with small singular values, the truncated SVD (TSVD) keeps only the first k components and discards the rest:

\hat m_k = \sum_{i=1}^{k} \frac{u_i^{\mathsf T}d}{\sigma_i}\,v_i.

Everything past index k — where the singular values are tiny and the Picard ratio has turned up — is dropped entirely. No division by near-zero, no noise blow-up. You trade a little bit of exact-fit-to-the-data for an enormous gain in stability, and for an ill-posed problem that is very much the trade you want.

A hard cutoff vs Tikhonov's soft taper

TSVD is regularization with a brick-wall filter: its filter factors are f_i = 1 for i \le k and f_i = 0 beyond. Compare this with Tikhonov, whose factors \sigma_i^2/(\sigma_i^2+\alpha^2) taper off smoothly. They are two points on the same spectrum: both suppress small-singular-value terms; TSVD does it with a switch, Tikhonov with a dimmer.

The truncation level k is the discrete cousin of \alpha — the dial that trades resolution against stability. Too small a k over-smooths (loses detail); too large lets the noise back in.

Switch vs dimmer

The bold step is the TSVD filter — 1 up to the cutoff k, then 0. The faint curve is the matching Tikhonov filter. Slide k: the wall moves, keeping more or fewer components. Both keep the strong, trustworthy directions and bin the weak ones — they differ only in how abruptly they make the cut.

Keep only the first k SVD components: \hat m_k = \sum_{i \le k}(u_i^{\mathsf T}d/\sigma_i)v_i.
Filter factors are a hard step (1 then 0) — Tikhonov is the smooth version.
k plays the role of \alpha: it trades resolution against stability.

Worked example — reconstructing with the top k components

Say the SVD hands you singular values \sigma_1 = 5,\ \sigma_2 = 2,\ \sigma_3 = 0.6,\ \sigma_4 = 0.02,\ \sigma_5 = 0.003, and the noise floor sits around 10^{-2}. The first three components divide by healthy numbers; the last two divide by near-zero and would each multiply the noise by 50 and by 330. So you set k = 3 and build

\hat m_3 = \frac{u_1^{\mathsf T}d}{5}\,v_1 + \frac{u_2^{\mathsf T}d}{2}\,v_2 + \frac{u_3^{\mathsf T}d}{0.6}\,v_3.

The two poisonous terms are simply gone. Your reconstruction is a little blurrier than the "true" answer would have been — you gave up the fine detail those last directions carried — but it is stable, and stability is what an ill-posed problem cannot otherwise give you.

Worked example — the k that is too small and the k that is too large

Imagine reconstructing a signal that genuinely has five meaningful features, buried in noise that overwhelms components 8 and beyond. Sweep the cutoff:

k = 2 (too small): you keep only the two broadest trends. The reconstruction is smooth and stable — but three real features have been smeared away. You have over-regularized: thrown out signal along with the noise.
k = 6 (about right): all five real features survive, the noisy tail is still excluded. This is the sweet spot.
k = 10 (too large): you have let components 8, 9, 10 back in — pure noise, amplified by tiny singular values. The reconstruction is jagged and wild. You have under-regularized.

This is the classic under/over trade-off in miniature, and picking the balance point is its own subject — choosing the regularization parameter.

Where to put the wall — read it off the Picard plot

You do not have to guess k blindly. Plot the Picard ratio |u_i^{\mathsf T}d|/\sigma_i against the index i. For the first several components the data coefficients |u_i^{\mathsf T}d| decay faster than the singular values, so the ratio stays flat or falls — those are the reliable directions. Then, at some index, the data stops decaying (it has hit the noise floor) while the singular values keep shrinking, and the ratio turns sharply upward. That turning point is your cutoff k: keep everything before the data stops decaying, drop everything after.

It is tempting to think there is one "correct" k the algorithm can hand you automatically. There isn't — choosing the truncation level is a genuine judgment call with no free lunch. Cut too aggressively (small k) and you smear away the very features you were trying to recover; cut too timidly (large k) and noise floods straight back in.

The Picard plot points you at a sensible neighbourhood, but the final k is a regularization parameter that must be tuned — with the discrepancy principle, cross-validation, or the L-curve — exactly like Tikhonov's \alpha. Treating k as a value you can look up rather than a knob you must set is the classic beginner's mistake. See choosing the regularization parameter.

Truncated SVD is regularization by editing out the untrustworthy directions — and that exact idea, "keep the big singular values, drop the small ones," is how SVD-based image and data compression works. A grayscale image is a matrix; its SVD sorts the image into layers by importance. Keep the top few dozen singular components and you reconstruct a picture that looks essentially identical to the original while storing a fraction of the numbers — the discarded small-singular-value layers were mostly fine texture and noise your eye never misses.

So denoising an inverse-problem reconstruction and shrinking a photo are, mathematically, close cousins: both keep the loud, reliable structure and quietly bin the faint, unreliable remainder. One calls the result "regularized," the other calls it "compressed."