Autocovariance and Autocorrelation

A stationary series has forgotten what year it is — so the only thing left to describe is how strongly a value is tied to its own past. Is today's temperature nearly a copy of yesterday's, or already half-forgotten? Does a shock echo for weeks or vanish overnight? The answer is a single function of the gap between two times, and it turns out to be the entire signature of a stationary process. Two series with the same autocorrelation function are, to second-order eyes, indistinguishable — it is the fingerprint of dependence.

This one function is the object every model in this course is secretly trying to reproduce, and the object every diagnostic plot is secretly displaying. Get comfortable with it here and the rest of the subject reads like commentary on it.

From covariance to a function of lag

Recall ordinary correlation between two variables. Now apply it to a series and its own shifted self. For a stationary process with mean \mu, the autocovariance at lag h is

\gamma(h) = \operatorname{Cov}(X_t, X_{t+h}) = \mathbb{E}\big[(X_t - \mu)(X_{t+h} - \mu)\big].

Stationarity is what makes this legal: the covariance depends on the lag h alone, not on where t sits, so \gamma is a function of one integer. Dividing by the lag-zero value strips out the units and rescales to a pure, dimensionless number — the autocorrelation function (ACF):

\rho(h) = \frac{\gamma(h)}{\gamma(0)}.

Because \gamma(0) = \operatorname{Var}(X_t), the ACF is just the correlation between X_t and X_{t+h}. At h = 0 a series is perfectly correlated with itself, so \rho(0) = 1 always; as h grows, \rho(h) tells you how much memory survives that many steps.

The properties that pin it down

The autocovariance function is not an arbitrary sequence of numbers — it must obey a short list of structural laws, and these are worth knowing because they are exactly the sanity checks you apply to any estimated ACF.

Non-negative variance: \gamma(0) = \operatorname{Var}(X_t) \ge 0, and it is the largest the function ever gets: |\gamma(h)| \le \gamma(0).
Symmetry (even function): \gamma(h) = \gamma(-h), so \rho(h) = \rho(-h) — looking forward a lag or backward a lag gives the same covariance.
Bounded correlation: |\rho(h)| \le 1 for all h, with \rho(0) = 1.
Positive semi-definiteness: for any times and any constants a_1, \dots, a_n, \sum_{i}\sum_{j} a_i a_j\, \gamma(t_i - t_j) \ge 0. This is the deep one — it is equivalent to \gamma being a valid autocovariance, and it is why not every symmetric decaying sequence can be an ACF.

The symmetry is why the ACF is conventionally drawn only for h \ge 0 — the negative side is a mirror image. And positive semi-definiteness is the fingerprint's DNA: it is the precise condition (via Bochner's theorem) that lets the same information reappear as a spectral density later in the course.

The shape of memory

A single family already shows the two archetypes you will meet everywhere: \rho(h) = \phi^{|h|}. Drag \phi and read the story off the curve. Positive \phi gives a smooth exponential taper — slow-fading memory, long swings. Negative \phi makes the ACF alternate in sign, the signature of a series that zig-zags. Near \pm 1 the memory stretches over many lags; near 0 it collapses after a single step — almost no memory at all.

This exact shape is the ACF of a first-order autoregressive process — a preview of how a model's parameter is read directly off the fingerprint.

Worked example: the ACF of an MA(1)-style toy

Take the little process X_t = \varepsilon_t + \theta\,\varepsilon_{t-1}, where the \varepsilon_t are uncorrelated, mean zero, variance \sigma^2 — a value is this instant's shock plus a fraction \theta of last instant's. Compute the autocovariances directly.

Lag 0: \gamma(0) = \operatorname{Var}(\varepsilon_t + \theta\varepsilon_{t-1}) = \sigma^2 + \theta^2 \sigma^2 = (1 + \theta^2)\sigma^2, since the two shocks are uncorrelated.
Lag 1: \gamma(1) = \operatorname{Cov}(\varepsilon_t + \theta\varepsilon_{t-1},\ \varepsilon_{t+1} + \theta\varepsilon_{t}). The only shared shock is \varepsilon_t, contributing \theta\,\sigma^2, so \gamma(1) = \theta\sigma^2.
Lag ≥ 2: no shock is shared, so \gamma(h) = 0 for |h| \ge 2.

Dividing through, the autocorrelations are

\rho(1) = \frac{\theta}{1 + \theta^2}, \qquad \rho(h) = 0 \ \text{ for } |h| \ge 2.

Put numbers in: with \theta = 0.5 and \sigma^2 = 1 you get \gamma(0) = 1.25, \gamma(1) = 0.5, and \rho(1) = 0.5 / 1.25 = 0.4. The ACF is one non-zero spike and then nothing — a sharp cut-off, the exact opposite of the autoregressive taper above. That contrast — gradual decay versus abrupt cut-off — is how you will tell the two model families apart from a picture.

Two traps ride along with the ACF. First, a large \rho(h) means the series predicts itself at that lag — it says nothing about an outside cause; a heater and an ice-cream van both correlate with the hour of day without either causing the other. Second, and subtler: \rho(h) is a marginal correlation that includes all the indirect paths through the intervening times. If today depends on yesterday and yesterday on the day before, then today and the day-before will show correlation even if there is no direct link — the dependence leaks through the chain. Stripping out those indirect echoes to expose the direct lag-h link needs a different tool, the partial autocorrelation function. Read the ACF as "total memory," not "direct memory."

Here is the reason you cannot just doodle any decaying wiggle and call it an autocorrelation function. Any real combination of the series, Y = \sum_i a_i X_{t_i}, is itself a random variable, and a variance can never be negative: \operatorname{Var}(Y) = \sum_i \sum_j a_i a_j\, \gamma(t_i - t_j) \ge 0. That single unavoidable fact — variances are non-negative — is positive semi-definiteness written out. It forbids, for instance, an ACF that is 1 at lag 0 and -1 at both lags 1 and 2 (you can build a combination with negative variance). So the property is not a technicality bolted on for rigour; it is the shadow cast by the simplest law of probability, and it is exactly what guarantees a matching non-negative spectral density exists.