Nonlinear Inverse Problems

Everything so far leaned on one comfortable assumption: the forward map was linear, a matrix A turning model into data, with a whole toolkit of pseudoinverses and clean formulas waiting to invert it. But nature rarely reads the memo. In an enormous number of real problems the data depends on the unknowns in a genuinely curved, tangled way.

Take seismology: how long a wave takes to travel from an earthquake to a station depends on the subsurface velocity — but the wave also bends along a path that itself depends on the velocity, so doubling a velocity somewhere does not simply double anything. The map from "velocity everywhere" to "arrival times" is nonlinear. This breaks the clean linear machinery: there is no matrix to invert, no single formula. Instead we must iterate — creep toward the answer one improving step at a time.

The challenge: no pseudoinverse, only iteration

Write the nonlinear forward model as d = G(m), where G is now a function, not a matrix (seismic travel times, electrical resistivity, chemical kinetics). Superposition fails and there is no closed-form inverse. So we change the question entirely: instead of solving for m, we minimize a data-misfit function

\Phi(m) = \tfrac12\,\lVert d - G(m) \rVert^2

— the total squared disagreement between what we measured and what a candidate model predicts. The best model is wherever \Phi is smallest. We hunt for that minimum iteratively, repeatedly linearizing the forward map around the current guess and taking a step downhill. This is exactly gradient-based optimization: nonlinear inversion is nonlinear least squares.

Linearize with the Jacobian

Around a current guess m_k, a first-order Taylor expansion replaces G by its Jacobian J_{ij} = \partial G_i/\partial m_j — the matrix of sensitivities of each datum to each parameter:

G(m_k + \delta m) \approx G(m_k) + J\,\delta m.

Fitting the leftover misfit \Delta d = d - G(m_k) is now a linear least-squares problem for the update \delta m, solved with the very tools from earlier — normal equations plus regularization. Update, recompute the Jacobian, repeat:

m_{k+1} = m_k + (J^{\mathsf T}J + \alpha^2 I)^{-1} J^{\mathsf T}\,\Delta d.

This is the regularized Gauss–Newton iteration. The Levenberg–Marquardt method adapts the damping \alpha as it goes — big when far from the solution (safe, gradient-descent-like small steps), small when close (fast, Newton-like leaps).

Nonlinear forward model d = G(m): no closed-form inverse; superposition fails.
Recast as minimizing the misfit \Phi(m) = \tfrac12\lVert d - G(m)\rVert^2.
Linearize around the current model with the Jacobian J = \partial G/\partial m.
Solve the regularized linear update (Gauss–Newton / Levenberg–Marquardt) and repeat to convergence.
Nonlinearity brings the danger of local minima — a good starting model matters.

Worked example — one turn of the loop

The iterative recipe, concretely:

Start with an initial guess m_0 (often from a simple model or prior knowledge).
Predict and compare: run the forward model to get G(m_k), form the misfit \Phi(m_k) and the residual \Delta d = d - G(m_k).
Sense: compute the Jacobian J — the gradient of the misfit is -J^{\mathsf T}\Delta d.
Update: solve the regularized linear system for \delta m and step to m_{k+1} = m_k + \delta m.
Repeat until the misfit stops shrinking meaningfully — that is convergence.

Simple enough. But there is a catch that never troubles the linear case, and it is worth its own picture.

Worked example — why the starting guess decides everything

A linear least-squares problem has a single, bowl-shaped misfit with one bottom: wherever you start, you slide to the same global optimum. Nonlinear problems are not so kind. Their misfit landscape can have several valleys — multiple local minima — and downhill steps only ever take you to the nearest one.

In the landscape above, a guess that starts on the left rolls into the left valley; a guess on the right rolls into the right valley — and only one of them is the true global minimum. Same data, same algorithm, different starting point, completely different answer. That is why the starting model matters so enormously in nonlinear inversion, and why it never mattered at all before.

In the linear case, least squares hands you the one global optimum and you can relax. Nonlinear inversion offers no such comfort. Because the misfit can have many local minima, different starting guesses can converge to completely different "solutions" that all fit the data reasonably well. The result you get can depend on where you started — a deeply uncomfortable fact if you were hoping for a single objective truth.

This is why practitioners treat nonlinear reconstructions with extra skepticism. They run multiple starts from different initial models to see whether answers agree, impose physical constraints (positivity, smoothness, known bounds) to fence off nonsense valleys, and report the spread of outcomes rather than a single confident number. A nonlinear "solution" quoted with no mention of the starting model or of multiple restarts deserves a raised eyebrow.

Each column of the Jacobian J is the sensitivity of every datum to one parameter. The naive way to get it — nudge parameter j, rerun the whole forward model, see what changed, and repeat for all parameters — costs one expensive simulation per unknown. With millions of unknowns that is hopeless.

The trick that makes large-scale nonlinear inversion possible is the adjoint method: by running the forward model once and a single "adjoint" (time-reversed) model once, you get the entire gradient J^{\mathsf T}\Delta d in two simulations instead of millions — independent of the number of parameters. It is the same idea as backpropagation in neural networks (which is exactly reverse-mode automatic differentiation), and it is why both fields can optimize over staggeringly many unknowns at all.

Nonlinear inverse problems are where inverse theory meets its hardest real-world tests. Full-waveform inversion tries to image the Earth from the full wiggles of seismic recordings — not just when a wave arrives, but the entire shape of the recorded waveform. It is so violently nonlinear, and involves so many parameters, that it devours some of the largest supercomputers on Earth. The oil and gas industry runs it for weeks on machines with hundreds of thousands of cores to map reservoirs kilometres underground.

And here is the twist that delights people: training a neural network is itself a giant nonlinear inverse problem. The "data" are the input–output examples, the forward map is the network, and the "model" you are recovering is the enormous vector of weights that reproduces the observed behaviour. Gradient descent, local minima, sensitivity to initialization, the whole family of worries — modern machine learning is inverse theory wearing a new hat, at a staggering scale.

Linear vs nonlinear, side by side

Solution: linear gives a one-shot formula (the pseudoinverse); nonlinear needs an iterative loop with no closing formula.
Misfit shape: linear is a single convex bowl (one global minimum); nonlinear can be a rugged landscape with many local minima.
Uniqueness: linear least squares finds the optimum; nonlinear may find a nearby one, depending on where you started.
Cost: linear is one solve; nonlinear pays for a forward run and a Jacobian (or gradient) at every iteration — sometimes thousands of them.

The linear machinery never disappears, though — it lives inside each nonlinear step, because every iteration is just a regularized linear least-squares problem for the update. Nonlinear inversion is linear inversion, done over and over, on a moving target.