Model Selection (AIC and BIC)

The estimation machinery will happily fit an ARMA(1,1), an ARMA(3,2), an ARMA(5,5) — any order you name. So which one do you keep? You cannot just pick "the one that fits best", because a bigger model always fits the training data at least as well: add a parameter and the likelihood can only go up. Chase that and you end up with a monster that memorises the noise in your sample and forecasts the future terribly. Model selection is the discipline of trading off fit against complexity — and the two tools everyone reaches for are the information criteria AIC and BIC.

The information criteria

Each criterion takes the maximised log-likelihood \hat\ell (higher = better fit) and docks it a penalty for the number of parameters k (here k = p + q + 1, counting the shock variance). You then pick the model with the smallest criterion.

Akaike information criterion: \mathrm{AIC} = -2\hat\ell + 2k.
Small-sample corrected: \mathrm{AICc} = \mathrm{AIC} + \dfrac{2k(k+1)}{T-k-1} — use it whenever T/k is small (say under 40); it converges to AIC as T \to \infty.
Bayesian information criterion: \mathrm{BIC} = -2\hat\ell + k\log T.

The shared shape is −2×fit + penalty×complexity. The -2\hat\ell term rewards fit; the penalty punishes extra parameters. All that separates the criteria is how steeply they charge for complexity.

The penalty is the whole story: AIC vs BIC

Look at the two penalties side by side: AIC charges 2 per parameter, BIC charges \log T. As soon as the sample exceeds T = 8 we have \log T > 2, so BIC penalises complexity more harshly — and the gap widens with T. The consequences are systematic:

Property	AIC	BIC
Penalty per parameter	2	\log T
Tends to pick	larger models	smaller, more parsimonious models
Best for	prediction / forecasting	finding the "true" order
Consistency	not consistent (may over-fit)	consistent (finds true model as T \to \infty)

This is the bias–variance trade-off wearing a different hat. Too few parameters and the model is biased — it cannot represent the truth. Too many and its estimates are noisy — high variance, poor out-of-sample forecasts. The criteria automate the search for the sweet spot: parsimony, the simplest model consistent with the data.

AIC across model orders

In practice you fit a grid of candidate orders and plot each criterion against complexity. The picture is almost always a U (or hockey-stick): the criterion falls steeply as the first few needed terms are added (fit improving fast), bottoms out at the right order, then creeps back up as extra parameters buy negligible fit for a growing penalty. The chart shows AIC and BIC over AR orders p = 0, \dots, 6: both bottom at p = 2, but BIC's steeper climb afterwards makes its minimum more decisive.

Reading the plot: pick the order at the bottom of the curve. When AIC and BIC disagree — AIC leaning toward a slightly larger model — favour BIC if you want a lean, interpretable model, and AIC if raw forecasting accuracy is the goal.

Worked example — choosing by ΔAIC

Two candidates on the same data:

Model A — ARMA(1,0), log-likelihood \hat\ell = -120, k = 2. Then \mathrm{AIC}_A = -2(-120) + 2(2) = 240 + 4 = 244.
Model B — ARMA(2,1), log-likelihood \hat\ell = -119, k = 4. Then \mathrm{AIC}_B = -2(-119) + 2(4) = 238 + 8 = 246.

Model B fits slightly better (log-likelihood up by 1), but at the cost of two extra parameters — and AIC judges that the improvement did not earn its keep. Since \mathrm{AIC}_A = 244 < 246 = \mathrm{AIC}_B, we choose the simpler Model A. The rule of thumb: a difference \Delta\mathrm{AIC} \lesssim 2 is weak evidence, while \Delta\mathrm{AIC} > 10 essentially rules the worse model out.

Automating the search: auto.arima

The whole procedure — fit every (p, d, q) in a sensible range, compute the criterion, keep the minimiser — is what automated routines like auto.arima do, wrapped around a Box–Jenkins workflow: they use unit-root tests to choose the differencing order d, then a stepwise or exhaustive search over (p, q) minimising AICc, and finish by checking the winner's residual diagnostics. Convenient, but not a licence to switch off your judgement — the criterion is one input, not the verdict.

The single most common misuse: comparing information criteria across models that are not on a level playing field. Two rules keep you honest. Same data. The criteria are only comparable when every model is fitted to the identical sample — same observations, same length. If one ARMA drops a few rows to make room for extra lags, its likelihood is computed over fewer points and its AIC is not comparable; align the samples first. Same degree of differencing. This is the big one for time series: you may not compare the AIC of a model on x_t with one on \nabla x_t. Differencing changes the response variable, so the likelihoods live on different scales entirely — the numbers are apples and oranges. Fix d first (via unit-root testing), then let AIC/BIC choose p and q among models sharing that same d.

BIC is consistent: if the true model really is a finite ARMA sitting in your candidate list, BIC will pick it with probability approaching one as T \to \infty. That sounds decisive — until you remember that the true model is almost never in your list. Real series are messy approximations, and here AIC's philosophy shines: it does not try to identify a "true" order but to minimise the expected forecasting error (the Kullback–Leibler divergence to the truth), which makes it efficient for prediction and better at picking a usefully rich model when the world is more complex than any candidate. So the honest answer to "AIC or BIC?" is "what is the job?": BIC to explain and find a parsimonious structure, AIC to forecast. Many analysts report both and worry when they disagree.