Linear Processes and the Wold Decomposition

Why should a handful of models — AR, MA, ARMA — be enough to describe almost every stationary series an analyst ever meets? The answer is a deep theorem, and it reads almost like a promise: every reasonable stationary process is white noise passed through a filter. Feed a stream of independent shocks into a linear system with a memory, and out comes trend-free structure, cycles, persistence — the entire zoo of stationary behaviour. Turn that sentence into mathematics and you get the linear process, and the theorem that guarantees it is Wold's decomposition.

A general linear process writes the series as a weighted sum of present and past white-noise shocks \varepsilon_t:

x_t = \sum_{j=0}^{\infty} \psi_j\,\varepsilon_{t-j} = \varepsilon_t + \psi_1 \varepsilon_{t-1} + \psi_2 \varepsilon_{t-2} + \cdots,

with \psi_0 = 1. This is an MA(\infty) — an infinitely long moving average. In backshift notation it is beautifully compact, x_t = \psi(B)\,\varepsilon_t, where \psi(B) = \sum_{j\ge 0}\psi_j B^{j}.

The one condition: square-summable weights

An infinite sum of random terms need not converge to anything sensible. The condition that keeps the process well-defined and stationary is that the weights be square-summable:

\sum_{j=0}^{\infty} \psi_j^{\,2} < \infty.

Intuitively, the influence of old shocks has to die away fast enough that the total variance stays finite. Under this condition the process has mean zero, finite variance, and — the useful part — an autocovariance you can write straight from the weights:

The variance is \gamma_0 = \sigma^2 \sum_{j\ge 0}\psi_j^{2}.
The lag-k autocovariance is \gamma_k = \sigma^2 \sum_{j=0}^{\infty} \psi_j\,\psi_{j+k}.
Hence the whole autocorrelation structure is determined by the \psi-weights — the weights are the memory.

So the shape of the \psi-sequence and the shape of the correlogram are two views of the same thing. Weights that decay slowly give long memory; weights that vanish after a few lags give an ACF that cuts off.

What the weights look like

Below, the \psi-weights follow the simplest decaying pattern, \psi_j = r^{\,j} — exactly the weights an AR(1) produces, where r = \phi. Drag r toward 1 and the tail stretches far into the past (long memory); pull it toward 0 and only the most recent shock matters (nearly white noise). As long as |r| < 1 the squared weights sum to 1/(1-r^2), comfortably finite.

Wold's decomposition

Herman Wold proved in 1938 that the linear-process form is not just a way to write a stationary series — for a huge class it is the way, and it is unique.

Any zero-mean covariance-stationary process x_t can be written as a sum of two uncorrelated parts, x_t = \sum_{j=0}^{\infty}\psi_j\,\varepsilon_{t-j} + v_t.
The first part is a linear process in white-noise innovations \varepsilon_t (the one-step-ahead prediction errors), with \psi_0=1 and \sum \psi_j^2 < \infty. This is the purely non-deterministic part.
The second part v_t is deterministic — perfectly predictable from its own infinite past (for example an exact sinusoid with a fixed amplitude).
The decomposition is unique, and the innovations are the residuals of the best linear forecast.

Read what this buys you. Strip off any perfectly predictable component, and whatever remains — no matter how it was generated — is an MA(\infty) driven by white noise. That is why so few model families suffice: to model a stationary series is to model its \psi-weights.

Why this makes ARMA so general

An MA(\infty) has infinitely many parameters — useless to fit directly. The engineering trick is to approximate the weight-generating function \psi(B) by a ratio of two short polynomials:

\psi(B) = \frac{\theta(B)}{\phi(B)} \quad\Longrightarrow\quad \phi(B)\,x_t = \theta(B)\,\varepsilon_t,

which is exactly an ARMA model. A rational function can mimic almost any decaying weight sequence to high accuracy with only a handful of coefficients. So Wold says "every stationary series is an infinite MA," and ARMA says "and I can capture that infinite MA with a few numbers." Together they are the theoretical backbone of the whole Box–Jenkins approach.

Wold's theorem is often misremembered as "every stationary series is Gaussian white noise filtered." It says no such thing. The innovations \varepsilon_t are only required to be white noise — uncorrelated, mean zero, constant variance — not independent and not normal. They can be wildly non-Gaussian, with dependence hiding in their higher moments (a GARCH volatility series is a perfect example: uncorrelated shocks that are absolutely not independent). The Wold representation captures a process only up to its second-order (covariance) structure. Two processes with the same \psi-weights share an ACF but can behave utterly differently. Don't read a linear representation as a claim about the full probability law.

The mysterious v_t in Wold's theorem is any component you could predict perfectly from its own infinite history with zero error — the classic example being a pure sinusoid A\cos(\omega t + \varphi) with fixed (if unknown) amplitude and phase. Sample it forever and you can extrapolate it exactly. Most real series, once trend and seasonality are removed, have no such component and are called purely non-deterministic — for them v_t = 0 and the series is a clean MA(\infty). The distinction matters in spectral analysis, where a deterministic sinusoid shows up as a sharp spike in the spectrum while the non-deterministic part spreads into a smooth density.