ARIMA Models

Nearly every headline series you actually care about — GDP, a share price, atmospheric CO₂, monthly airline traffic — has one inconvenient feature in common: it trends. It drifts up (or down) over the years, so its mean is not constant and its variance keeps growing. That is fatal for the machinery we have built so far, because ARMA models assume a stationary process — one whose statistical character does not change with time. Fit an ARMA straight onto a trending series and you get nonsense.

The ARIMA model is the elegant fix. Its bet is that although the level of the series wanders, its changes are well-behaved. So we difference the series — replace it by its step-to-step differences — as many times as it takes to flatten the trend, fit a tidy ARMA to the stationary result, and then undo the differencing to get back to the original scale. Three letters, three jobs: AR (autoregression), I (integrated), MA (moving average).

The model in one line

Write B for the backshift operator (Bx_t = x_{t-1}), so the first difference is \nabla x_t = x_t - x_{t-1} = (1-B)x_t. An ARIMA(p,d,q) process is then written with breathtaking compactness:

\phi(B)\,(1-B)^d\, x_t = \theta(B)\,\varepsilon_t,

where \varepsilon_t is white noise and the two polynomials carry the AR and MA parts:

\phi(B) = 1 - \phi_1 B - \dots - \phi_p B^{p}, \qquad \theta(B) = 1 + \theta_1 B + \dots + \theta_q B^{q}.

p — the autoregressive order: how many past values the (differenced) series leans on.
d — the degree of differencing: how many times we apply (1-B) to reach stationarity.
q — the moving-average order: how many past shocks feed into the present value.

Read it as a recipe: difference d times, then the result is an ARMA(p,q). Setting d=0 recovers an ordinary ARMA — every ARMA is an ARIMA with no differencing. The whole ARMA toolkit (ACF/PACF reading, estimation, forecasting) is inherited wholesale; the "I" is just a change of variable applied first.

Why "integrated"?

Differencing is a discrete derivative: \nabla x_t is the change per step. Its inverse is a discrete integral — a running sum. If you know the differences w_t = \nabla x_t and a starting value x_0, you rebuild the level by summing them back up:

x_t = x_0 + \sum_{j=1}^{t} w_j.

So the observed series is the accumulation — the integral — of a stationary ARMA process. That is exactly what "integrated of order d", written x_t \sim I(d), means: you must difference it d times to get something stationary, or equivalently, it is a d-fold running sum of a stationary series. A stationary series is I(0); a series with an ordinary trend is usually I(1).

The random walk x_t = x_{t-1} + \varepsilon_t rearranges to (1-B)x_t = \varepsilon_t.
That is \phi(B)=1 (no AR terms, p=0), d=1, \theta(B)=1 (no MA terms, q=0): pure ARIMA(0, 1, 0).
Difference it once and you are left with plain white noise — the simplest possible ARMA.

Choosing d: look before you difference

In practice d is almost always 0, 1, or 2. A series with a roughly linear trend needs d=1; a series whose slope itself drifts (a trend in the trend) may need d=2; genuinely seasonal wandering is handled separately by seasonal differencing. You rarely need more — over-differencing does real harm (see the warning below).

The picture tells the story. Below is a series that trends steadily upward: its mean is clearly not constant, so it is not stationary and ARMA cannot touch it.

Now take one difference — plot \nabla x_t = x_t - x_{t-1}, the change from each month to the next. The trend vanishes; what remains fluctuates around a fixed level with roughly constant spread. That is what we fit an ARMA to.

The formal decision — "is that trend real enough to demand differencing, or could it be stationary noise?" — is settled by a unit-root test, the subject of the next lesson.

Worked example: naming the orders

Suppose you have monthly data with a clear upward trend. You take one difference and the trend disappears; the differenced series looks stationary. You then inspect its correlation structure and find a partial autocorrelation that cuts off after lag 1, and an autocorrelation that tapers off gradually. A PACF that cuts off at lag 1 with a decaying ACF is the classic fingerprint of an AR(1). So the differenced series is an ARMA(1, 0), and the original series is

\text{ARIMA}(1, 1, 0): \qquad (1 - \phi_1 B)(1 - B)\,x_t = \varepsilon_t.

Multiplying out, this says x_t depends on x_{t-1} and x_{t-2} — the differencing and the AR term combine into a two-lag rule on the original level. Reading the three orders straight off the differenced ACF/PACF is precisely the Box–Jenkins identification step.

If differencing once fixes the trend, it is tempting to think differencing twice makes things "even more stationary". It does the opposite. Difference a series that was already stationary and you inject a spurious negative correlation: the first difference of white noise is an \text{MA}(1) with a unit root in its MA polynomial — a non-invertible model that is numerically nasty to estimate and inflates the forecast variance. The symptom to watch for: the differenced ACF has a sharp spike near -0.5 at lag 1 and the variance has gone up, not down. Rule of thumb — difference the fewest times that removes the trend, and never difference just because you can.

The most widely used forecasting heuristic in industry — simple exponential smoothing, where your forecast is a weighted average of the past that decays geometrically — turns out to be the optimal forecast of a specific ARIMA model: the IMA(1, 1), i.e. ARIMA(0, 1, 1), (1-B)x_t = (1 + \theta_1 B)\varepsilon_t. The smoothing constant is just a repackaging of \theta_1. Two communities — engineers tuning a smoothing dial and statisticians estimating an MA coefficient — were fitting the very same model for decades without realising it. Recognising practical recipes as special cases of ARIMA is one of the quiet triumphs of the Box–Jenkins framework.