Confidence Intervals

On election night a pollster announces: "43% support the measure." It sounds precise. But they only asked a thousand people, not the whole country — ask a different thousand tomorrow and you might get 41%, or 45%. So the single number 43\% is a little bit of a lie: it hides how shaky it is.

The honest version says a range instead: "43% \pm 3%, and we're 95% confident the true figure lies between 40% and 46%." That is a confidence interval. It takes one fragile guess and turns it into an interval with a stated reliability — and it is exactly how every poll, drug trial, and careful scientific measurement reports its uncertainty. A number without an interval is a rumour; a number with one is a measurement.

The anatomy of an interval

A point estimate is a single dot with no sense of its own error. A confidence interval fixes that by wrapping the dot in a margin:

\underbrace{\hat{\theta}}_{\text{estimate}} \;\pm\; \underbrace{z^{\*}\cdot \mathrm{SE}}_{\text{margin of error}}

Three pieces do all the work:

The half-width z^{\*}\cdot\mathrm{SE} is the margin of error — the \pm 3\% you always hear quoted. Why "about two standard errors"? The empirical rule: about 95\% of a bell curve sits within two standard deviations of its centre, and the sampling distribution of the estimate is (near enough) a bell curve.

Worked example: build a 95% interval

A survey estimates support at \hat{p} = 0.43 with a standard error of \mathrm{SE} = 0.015. Build the 95\% interval with the quick 2-SE rule.

  1. Margin of error: 2 \times 0.015 = 0.03, i.e. three percentage points.
  2. Reach out both ways from the estimate: 0.43 - 0.03 = 0.40 and 0.43 + 0.03 = 0.46.
  3. State it: the 95\% confidence interval is [0.40,\ 0.46] — "43% \pm 3%".

In plain English: the method we used produces intervals that capture the true level of support about 95% of the time, and for this sample that interval runs from 40% to 46%.

Same data, but now we want to miss less often. The critical value climbs from z^{\*}\approx 1.96 to z^{\*}\approx 2.58, so the margin grows to 2.58 \times 0.015 \approx 0.039 and the interval widens to about [0.391,\ 0.469]. More certainty costs width: to be surer of catching the truth, you have to cast a wider net. There is no free lunch — you buy confidence with vagueness.

Worked example: a bigger sample tightens the net

Suppose we quadruple the sample. The standard error shrinks like 1/\sqrt{n}, so four times the data halves the SE — from 0.015 down to 0.0075. Redo the 95\% interval:

  1. New margin: 2 \times 0.0075 = 0.015, just 1.5 points.
  2. New interval: 0.43 \pm 0.015 = [0.415,\ 0.445].

Same confidence level, same centre — but the interval is half as wide. This is the whole reason big polls cost money: precision is bought with sample size, and because of the \sqrt{n}, buying it gets expensive fast (four times the people for twice the precision).

What the 95% really describes

Here is the subtle part. A 95\% confidence interval is built by a procedure with this property: if you repeated the whole experiment many times — new sample, new interval each time — then about 95\% of those intervals would capture the true \mu. The confidence lives in the recipe, not in any one interval it produces.

So it is tempting but wrong to say "there is a 95\% chance \mu is in this interval". Once computed, your interval either contains the fixed \mu or it does not — there is no probability left in it. The 95\% is the long-run hit rate of the method.

A stack of intervals, one true μ

The vertical line is the fixed true \mu. Each horizontal bar is one confidence interval from one sample, centred on that sample's estimate. Most bars cross the line — they captured \mu. The odd one out (drawn in a warning colour) sits entirely to one side: it missed. Over many intervals, the fraction that catch \mu hovers near the confidence level.

Asking for more confidence means missing less often — and the only way to miss less often with the same data is to make every interval wider. Confidence and width pull in the same direction.

It is desperately tempting to read "95% confidence" as "there is a 95% probability the true value lies in this interval." That sentence is wrong, and it is wrong in a deep way, not a nit-picking one.

The true value \mu is a fixed number, not a random one. Your interval [0.40, 0.46] is also fixed once you've computed it. A fixed number is either inside a fixed interval or it isn't — the probability is already 0 or 1, we just don't know which. There is no "95% chance" left to talk about.

The correct reading puts the randomness where it belongs, in the sampling procedure: "if we repeated this whole sampling-and-interval-building process many times, about 95% of the intervals we'd build would contain the true value." The confidence is a property of the method, not of your one particular interval. Say "95% of intervals like this one capture \mu", never "this interval has a 95% chance."

There is nothing sacred about 95\%. Ronald Fisher, sorting out modern statistics in the 1920s, casually suggested that landing within two standard deviations was a convenient line to draw — and two standard deviations happens to fence off about 95\% of a bell curve. The convention stuck out of sheer habit, not because the universe cares about the number 95.

What the convention really encodes is a bargain. Push the confidence all the way to 100\% and watch what happens: to be absolutely certain of catching the true value, your interval would have to stretch from -\infty to +\infty — guaranteed correct, and utterly useless. So every confidence interval is a deliberate compromise: narrow enough to be informative, wide enough to usually be right. Choosing 95% is just choosing where on that trade-off to stand.

See it explained