A physicist reports a particle's mass as 938.3 \pm 0.5 MeV. A pollster reports 43% support. In both cases the raw material is a fistful of noisy numbers, and the job is to say something disciplined about the process that produced them. Mathematical statistics is the theory of exactly that step — turning data into statements about an unknown mechanism — and it rests on two objects: a model for how the data could have arisen, and an estimator that reads a parameter of that model off the data.

This page pins down both, together with the three numbers we use to judge an estimator: bias, variance, and the mean squared error that binds them. It is the grammar the rest of the module is written in.

A statistical model is a family of distributions

We treat the data X_1,\dots,X_n as a random sample — one draw from some probability distribution we cannot see. A parametric statistical model is a whole family of candidate distributions, indexed by a parameter \theta that ranges over a parameter space \Theta:

The modelling assumption is that some true \theta_0 generated the data — we just don't know which. Examples: heights are N(\mu,\sigma^2) with \theta=(\mu,\sigma^2) and \Theta=\mathbb{R}\times(0,\infty); a stream of yes/no answers is \text{Bernoulli}(p) with \Theta=[0,1]; hourly arrival counts are \text{Poisson}(\lambda) with \Theta=(0,\infty).

Statistic vs. parameter — the fault line

Everything downstream depends on keeping two kinds of quantity apart.

• A parameter \theta is a fixed (usually unknown) feature of the distribution — \mu, \sigma^2, p. It is a property of the world, not of your sample.
• A statistic T = T(X_1,\dots,X_n) is any quantity computed from the data alone. Because it is a function of random data it is itself a random variable, with its own distribution — the sampling distribution.

An estimator is simply a statistic chosen to guess a parameter; we write \hat\theta = T(X_1,\dots,X_n). A specific number it produces on one dataset is an estimate. The crucial and easily-missed point: an estimator is random. Draw a fresh sample and it jumps. So we do not ask "is this one estimate right?" — we ask about the behaviour of the whole sampling distribution.

Bias, variance, and mean squared error

Three numbers summarise how good an estimator \hat\theta is for a true value \theta. The bias measures systematic lean:

An estimator with zero bias for every \theta is called unbiased: on average, over all the samples we might have drawn, it lands exactly on target. The variance \operatorname{Var}_\theta(\hat\theta) measures how much it scatters from sample to sample. The single figure that captures overall accuracy is the mean squared error, and it splits cleanly into the two:

This is the bias–variance decomposition. It says accuracy has two enemies — scatter and lean — and a good estimator keeps both small. Sometimes we even accept a little bias to buy a large cut in variance, because MSE is what actually costs us.

Worked example 1 — the sample mean is unbiased

Let X_1,\dots,X_n be i.i.d. with mean \mu and variance \sigma^2. Take the estimator \hat\mu = \bar X = \tfrac1n\sum_i X_i. By linearity of expectation,

so \bar X is unbiased for \mu. Its variance shrinks with the sample size:

Because it is unbiased, its MSE equals that variance, \sigma^2/n — pure scatter, no lean, and it vanishes as n\to\infty.

Worked example 2 — why we divide by n-1

The "obvious" variance estimator \hat\sigma^2_n = \tfrac1n\sum_i (X_i-\bar X)^2 is biased: because the deviations are measured from the sample mean \bar X rather than the true \mu, it systematically underestimates. A calculation gives

Dividing by n-1 instead of n exactly corrects the lean, giving the unbiased sample variance s^2 = \tfrac{1}{n-1}\sum_i (X_i-\bar X)^2 with \mathbb{E}[s^2]=\sigma^2. That "minus one" is Bessel's correction, and it is a bias fix, nothing more mysterious.

See the decomposition move

The vertical line marks the true value \theta. The bell is the estimator's sampling distribution. Slide bias to shift its centre off the true value, and spread to widen it. Watch the readout: the mean squared error is always the variance plus the squared bias — push either up and MSE climbs.