Autoregressive Models

Here is the oldest and most intuitive idea in time-series modelling: to predict where a series is going, look at where it just was. Tomorrow's temperature is a nudge away from today's. This month's unemployment is last month's plus a small change. An autoregressive model turns that instinct into an equation — the series is regressed on its own past, hence auto-regressive.

The first-order model, AR(1), is the whole idea in miniature:

x_t = \phi\, x_{t-1} + \varepsilon_t,

where \varepsilon_t is fresh white noise — an unpredictable shock with mean zero and variance \sigma^2 — and \phi is a single number that controls the memory. Each value is a fraction \phi of the previous value, plus a random kick. That is it. From this one line spills a startling variety of behaviour.

What φ does

The parameter \phi is the persistence dial. Read off its three regimes:

0 < \phi < 1: shocks fade smoothly. The series hovers around its mean and drifts back after a push — mean reversion. Close to 1 it clings to its past and wanders in long, slow swings; close to 0 it is almost pure noise.
-1 < \phi < 0: the series alternates — a high value tends to be followed by a low one, zig-zagging around the mean.
|\phi| \ge 1: shocks grow instead of fading. The series is no longer stationary — at \phi = 1 it becomes the random walk, which never forgets a shock at all.

So stationarity of an AR(1) requires |\phi| < 1 — the pendulum must lose energy between kicks, or it flies apart.

The mean, variance and autocorrelation

Assume |\phi| < 1 and take expectations of both sides: with a zero-mean shock the process has mean zero (add a constant term c and the mean becomes c/(1-\phi)). Squaring and taking expectations gives the variance,

\gamma_0 = \operatorname{Var}(x_t) = \frac{\sigma^2}{1 - \phi^2},

which blows up as \phi \to 1 — persistent series swing ever wider. The prize result is the autocorrelation function. Multiply the defining equation by x_{t-k}, take expectations, and a one-line recursion gives a clean geometric answer:

The autocorrelation at lag k is \rho_k = \phi^{\,k} for k \ge 0.
It decays geometrically — a smooth exponential taper when \phi > 0, an alternating decay when \phi < 0.
It never hits exactly zero: an AR process has infinite memory that fades but never ends.

That slowly-tapering ACF is the visual fingerprint of an autoregressive series — and it is exactly what distinguishes AR from the moving-average models, whose ACF cuts off abruptly.

The ACF, live

Drag \phi and watch the autocorrelation \rho_k = \phi^k as a function of lag. Push it toward 1 and the memory stretches out to long lags; pull it toward 0 and correlation collapses after a single step; take it negative and the bars flip sign lag by lag.

Higher order: AR(p)

Let the series lean on more of its past and you get the order-p model:

x_t = \phi_1 x_{t-1} + \phi_2 x_{t-2} + \dots + \phi_p x_{t-p} + \varepsilon_t.

Yule's sunspots need p = 2: an AR(2) can produce pseudo-cycles — oscillations with a preferred period that drift in phase, exactly the ragged near-periodicity of real cyclic data, with no hidden sine wave required. The coefficients that make an AR(p) stationary are pinned down by the roots of its characteristic polynomial (the topic of causality and invertibility), and they are estimated from data by the Yule–Walker equations. How many lags p to include is read off the partial autocorrelation function, which cuts off cleanly at lag p.

When you estimate an AR(1) and get \hat\phi = 0.98, it is tempting to treat it as a (barely) stationary, mean-reverting series. Be careful: a genuine random walk (\phi = 1) fitted on a finite sample will also hand you an estimate just below one, because the estimator is biased downward near the boundary. The difference between "strongly mean-reverting" and "not stationary at all" is enormous for forecasting — the first pulls back to a fixed mean, the second drifts forever — yet the two look nearly identical in a fitted coefficient. You cannot settle it by eyeballing \hat\phi; you need a unit-root test designed exactly for this knife-edge.

The mechanical picture that unlocked the subject: imagine a damped pendulum. Left alone it swings and slowly settles — that is the deterministic part, \phi x_{t-1} (or a two-term version for AR(2)). Now let a mischievous child kick it at random moments — that is the noise \varepsilon_t. The pendulum never settles and never repeats exactly, yet it keeps a characteristic swing period set by its physics. Yule realized the sunspot cycle behaves just like this: no celestial metronome, just a system with a natural frequency being jostled by chance. Autoregression is the mathematics of kicked pendulums — which is why it shows up everywhere from economics to climate to your own heartbeat.