Ergodicity

We have arrived at the quiet miracle that makes the whole subject work. A stochastic process defines its mean and its dependence as ensemble averages — quantities you would compute by averaging across infinitely many parallel realizations. But we only ever get one realization, one row of the ensemble table. How on earth can a single history teach us about an average over histories we never see?

The answer is ergodicity: the property that lets you swap an average across the ensemble for an average along time. When a process is ergodic, one long enough sample path visits its states in the right proportions, so the time average you can compute converges to the ensemble average you want. Every time you estimate a mean by \bar{x} or an autocorrelation by a correlogram, you are silently betting on ergodicity.

Time average versus ensemble average

Fix the idea with the two averages side by side. The ensemble average at time t is the expected value across realizations,

\mu = \mathbb{E}[X_t] \quad\text{(average down the column, over the ensemble)},

while the time average of your one observed path of length T is

\bar{X}_T = \frac{1}{T}\sum_{t=1}^{T} X_t \quad\text{(average along the row, over time)}.

These are conceptually completely different operations — one sweeps sideways across universes, the other marches forward through one universe. Ergodicity is precisely the statement that, for a stationary process, they nonetheless give the same answer in the limit.

A stationary process is mean-ergodic if the time average converges to the ensemble mean: \bar{X}_T \to \mu as T \to \infty (in mean-square).
A sufficient condition is that the autocovariances die away: \gamma(h) \to 0 as h \to \infty (more precisely, \frac{1}{T}\sum_{h} \gamma(h) \to 0).

The intuition is memory. If distant observations are nearly uncorrelated, a long series contains many effectively independent chunks, and averaging them behaves like the law of large numbers — \bar{X}_T homes in on \mu. If instead the correlation never fades, the whole series is really "one big correlated blob" and averaging it buys you almost nothing.

Convergence you can watch

The top figure shows the running time-average \bar{X}_T of a well-behaved stationary series whose correlations decay. It swings about early on — few observations, much wobble — then tightens onto the ensemble mean \mu = 0 as T grows. This is ergodicity happening: one path, given enough time, reveals the ensemble truth.

The bottom figure shows the process that breaks it, and it is worth staring at.

Three realizations of X_t = A + \varepsilon_t, where A is drawn once per path and frozen. Each running average converges beautifully — but to its own level A, not to the ensemble mean 0. No single path, however long, ever discovers that the ensemble mean is zero. Time-averaging fails to recover the ensemble average. That is non-ergodicity made visible.

The canonical counterexample, worked

Make the failure precise with X_t = A, where A \sim \mathcal{N}(0, \tau^2) is a single random constant drawn at the dawn of time and never changed (a degenerate case of the frozen-level process, with no noise). Check that it is stationary:

\mathbb{E}[X_t] = \mathbb{E}[A] = 0 — constant mean ✓;
\operatorname{Var}(X_t) = \tau^2 — constant variance ✓;
\operatorname{Cov}(X_t, X_{t+h}) = \operatorname{Var}(A) = \tau^2 for all h — depends only on lag (trivially) ✓.

So \{X_t\} is perfectly (even strictly) stationary. Now the time average of any single realization is

\bar{X}_T = \frac{1}{T}\sum_{t=1}^{T} A = A \quad\text{for every } T,

which equals the frozen A of that path — a random number that is almost never 0 = \mathbb{E}[X_t]. The time average does not converge to the ensemble mean; it converges to whatever this universe happened to draw. And the tell-tale is right there in the autocovariance: \gamma(h) = \tau^2 never decays — the memory is infinite and permanent, so the mean-ergodicity condition fails outright.

It is dangerously easy to treat "stationary" and "ergodic" as synonyms — most well-behaved textbook examples are both, so the distinction rarely bites in practice, and then one day it does. They are genuinely different properties. Stationarity is about the ensemble: the process's laws don't change over time. Ergodicity is about whether one path can recover those ensemble laws by time-averaging. The frozen-constant process above is the clean counterexample: impeccably stationary, yet a single realization is stuck at one level forever and can never reveal the ensemble mean. The practical danger is real — if the thing you are studying has a persistent, path-specific component (a permanent regime, an individual fixed effect that never averages out), then no amount of additional data from that one series will estimate the ensemble quantity you care about. More history does not help when history is non-ergodic. Stationarity lets the ensemble stay still; only ergodicity lets one path stand in for the ensemble.

It was coined by the physicist Ludwig Boltzmann in the 1870s, welding together the Greek ergon (work, energy) and hodos (path). He was wrestling with a foundational question of thermodynamics: a box of gas has astronomically many molecules, and we cannot possibly track them all — yet we blithely compute the pressure as an average over all possible microscopic states (an ensemble average) and it matches what a single gauge reads over time. The "ergodic hypothesis" was Boltzmann's bold claim that a single system, left long enough, wanders through essentially all its accessible states in the right proportions, so its time average equals the ensemble average. Statistical mechanics rests on it; time-series analysis borrowed both the idea and the name. When you compute a sample mean from one series and trust it, you are Boltzmann trusting his gauge — quietly assuming the one path you have is representative of all the paths you don't.