Smoothing and General Tikhonov
Imagine you are reconstructing the temperature down the length of a long metal rod from a handful
of noisy sensor readings. You already know a naive least-squares fit will explode into a spiky,
oscillating mess — that is what makes the problem
ill-posed.
So you reach for
Tikhonov regularization
to tame it. But standard Tikhonov penalises \|m\|^2 — the size
of the solution — and so it pulls your whole temperature profile towards zero.
That is bizarre: a rod sitting in a warm room has no reason to be near 0^\circ
everywhere. You didn't want a small answer. You wanted a smooth one.
General Tikhonov (also called generalized Tikhonov) fixes exactly this.
It swaps the penalty for the norm of a linear operator L applied to the
model:
\hat m = \arg\min_m \Big( \|Gm - d\|^2 + \alpha^2\,\|L m\|^2 \Big).
When L is a derivative operator, the penalty measures
how rough the solution is instead of how big it is. Now the cheap solutions are
the gently-varying ones, and the reconstruction comes out smooth rather than shrunken. Choosing
L is choosing what the word "simple" means — it quietly encodes your
prior belief about what a good answer looks like.
- L = I: penalise size → a small solution (this is standard Tikhonov, recovered as a special case).
- L = D_1 (first difference, a discrete first derivative): penalise slope → a flat solution.
- L = D_2 (second difference, a discrete second derivative): penalise curvature → a smooth solution.
What a difference operator actually is
The operator L is just a matrix that approximates a derivative by
subtracting neighbours. For a model sampled at points m_1, m_2, \dots, m_n,
the first-difference operator D_1 produces the gaps
between adjacent values,
(D_1 m)_i = m_{i+1} - m_i,\qquad
D_1 = \begin{pmatrix} -1 & 1 & & \\ & -1 & 1 & \\ & & \ddots & \ddots \end{pmatrix}.
So \|D_1 m\|^2 = \sum_i (m_{i+1}-m_i)^2 is small precisely when
neighbouring values are close — a flat profile. The second-difference
operator D_2 takes the difference of the differences,
(D_2 m)_i = m_{i+1} - 2m_i + m_{i-1},
which is a discrete curvature. Making \|D_2 m\|^2 small favours a curve
that bends as little as possible — a smooth profile, closer to a straight line or a gentle
arc. Same machinery, different L, different notion of "simple".
The penalty is a prior
The solution is again a modified normal equation. Setting the gradient of the objective to zero
gives
(G^{\mathsf T}G + \alpha^2 L^{\mathsf T}L)\,\hat m = G^{\mathsf T}d.
The extra \alpha^2 L^{\mathsf T}L term makes oscillatory, jagged
solutions expensive, so the minimiser comes out smooth — exactly suppressing the high-frequency
noise that ill-posedness amplifies. This is the same instinct as a smoothing prior, and it
foreshadows the
Bayesian reading:
the penalty operator L is the inverse square root of a prior covariance
— a precise statement of "I expect the answer to be smooth".
Three worked scenarios
Take the same forward problem Gm = d — noisy, ill-posed — and change
only the penalty operator. Nothing about the data changes; watch how the character of the
reconstruction flips.
1. L = I — penalise size
Solve (G^{\mathsf T}G + \alpha^2 I)\hat m = G^{\mathsf T}d. Every
component of \hat m is nudged towards 0. If
the true profile is a bump of height 50, standard Tikhonov gives you a
bump that is a bit too short — the whole thing is shrunk. Good when the
truth really is near zero and modest in size; wrong when it isn't.
2. L = D_1 — penalise slope
Solve (G^{\mathsf T}G + \alpha^2 D_1^{\mathsf T}D_1)\hat m = G^{\mathsf T}d.
Now steep changes between neighbours are expensive, so the reconstruction is pushed towards being
flat — a nearly constant level, or a staircase of gentle steps. The height is
not pulled to zero; a flat plateau at 50^\circ pays no first-difference
penalty at all, because all its neighbour-gaps are zero.
3. L = D_2 — penalise curvature
Solve (G^{\mathsf T}G + \alpha^2 D_2^{\mathsf T}D_2)\hat m = G^{\mathsf T}d.
Curvature is expensive, so the answer flattens towards a smooth gently-bending
curve. A straight ramp of any slope pays zero second-difference penalty, so linear trends survive
untouched while wiggles are ironed out. This is usually the best match for a physical profile like
temperature or density that varies continuously.
The moral in one line: standard Tikhonov shrinks toward zero; smoothing Tikhonov
flattens toward a smooth curve. Same solver, radically different answer, chosen entirely
by L.
Smooth vs jagged
Two candidate solutions that fit the data about equally well: a smooth one and a jagged one
carrying high-frequency wiggles. A first- or second-derivative penalty
\|Lm\|^2 is small for the smooth curve and large for the jagged one, so
general Tikhonov picks the smooth candidate — the wiggles are exactly the amplified noise we want
gone. Every jiggle in the orange curve has a steep slope and sharp curvature, so
\|D_1 m\|^2 and \|D_2 m\|^2 both blow up on it.
- Minimise \|Gm-d\|^2 + \alpha^2\|Lm\|^2; solve (G^{\mathsf T}G + \alpha^2 L^{\mathsf T}L)m = G^{\mathsf T}d.
- L = I → small; L = D_1 → flat; L = D_2 → smooth.
- The choice of L encodes a prior about the solution.
Digging deeper
The choice of L is not a neutral technicality — it is an
assumption about what the true answer looks like. Penalising roughness assumes
the truth really is smooth. So if the real profile has a genuine sharp jump — the edge
of a buried object, a material boundary, a step in density — a smoothness penalty will faithfully,
confidently, and wrongly blur it away into a gentle ramp. You will get back a
beautiful smooth curve that has quietly destroyed the one feature you were trying to find.
This is exactly why edge-preserving methods exist — total-variation regularization, for
instance, penalises \sum_i |m_{i+1}-m_i| (an L1 norm of the gradient)
instead of its square, which allows a few sharp jumps to survive while still killing the
small-scale noise. It is the standard tool for images with hard boundaries. The rule to burn in:
your regularizer must match your prior belief about the solution. Choose a
smoothness penalty only when you truly believe the answer is smooth.
Yes — and this is the beautiful part. Every regularizer is a disguised statement of
prior belief about the answer, before you ever look at the data:
- \|m\|^2 whispers "I expect the answer to be small."
- \|D_2 m\|^2 whispers "I expect the answer to be smooth."
- a total-variation penalty whispers "I expect sharp edges separating flat regions."
That is precisely the bridge to the Bayesian view of regularization, coming up in
priors are regularization,
where every regularizer turns out to correspond to a prior probability distribution over
possible solutions — and the mysterious knob \alpha becomes an exact
ratio of noise to prior confidence. The "arbitrary penalty" you added was a probability statement
the whole time.
Notice how little the machinery moves between standard and general Tikhonov: you only replace
\alpha^2 I with \alpha^2 L^{\mathsf T}L in
the normal equations. Everything else — the least-squares fit, the linear solve, the way
\alpha trades data-fit against penalty — is identical. That is the
quiet power of writing the penalty as \|Lm\|^2: one general form, and
you dial in "small", "flat", "smooth", or anything in between just by picking the matrix
L. You can even stack requirements, e.g.
L^{\mathsf T}L = \beta_0 I + \beta_2 D_2^{\mathsf T}D_2, to want a
solution that is both modestly-sized and smooth.