The Regression Line

Correlation tells you that two things move together — taller people tend to weigh more, more study tends to earn more marks. Useful, but it stops at "they're related." The regression line goes one crucial step further: it hands you a prediction machine.

Feed it any x and it returns your best estimate of y. Know a house's floor area? It'll guess the price. Know today's temperature? It'll guess the ice-cream sales. That single straight line, threaded through a whole cloud of scattered points, is the quiet engine behind almost every forecast made from data — from insurance premiums to climate models to the "customers also bought" box.

Written as a prediction for y from x, it is

\hat y = a + b x,

where the hat on \hat y marks it as a predicted value: b is the slope and a the intercept. But of all the lines we could draw through the cloud, which one is "best"?

Least squares

For each observation the line makes a prediction \hat y_i = a + b x_i, and the vertical gap to the actual point is the residual

e_i = y_i - \hat y_i.

A residual is positive when the point sits above the line, negative below. The regression line is the one that makes these gaps small overall — specifically, the line chosen by least squares: it minimises the sum of squared residuals

\text{SSE} = \sum_{i=1}^{n} \left(y_i - \hat y_i\right)^2 = \sum_{i=1}^{n}\left(y_i - a - b x_i\right)^2.

We square the gaps so positives and negatives cannot cancel, and so that a few large misses are penalised heavily. The slope and intercept that drive \text{SSE} to its smallest possible value are the regression line. That single quantity to be made small has a name you will meet everywhere in machine learning: a cost function. Fitting a line is the simplest possible case of "find the settings that make the cost smallest" — the same idea that, scaled up, trains enormous predictive models.

Hunt for the line

Drag the slope and intercept to move the line through the fixed scatter. The coloured stalks are the residuals — the vertical gaps from each point to the line — and the live \text{SSE} readout adds up their squares. Try to make it as small as you can; the line that wins is the least-squares regression line.

Three worked examples

1) Using the machine. A study fits \hat y = 1.2 + 0.8\,x where x is hours revised and y is the mark (out of 10). To predict the mark for a student who revised x = 6 hours, just substitute: \hat y = 1.2 + 0.8 \times 6 = 1.2 + 4.8 = 6.0. The line predicts a mark of 6 out of 10. That is all a prediction is — plug in x, read out \hat y.

2) Why "balanced". Suppose one real student who revised 6 hours actually scored 7. Their residual is 7 - 6.0 = +1.0 — they sit above the line. Another 6-hour student scored 5, a residual of 5 - 6.0 = -1.0, below the line. The least-squares line settles exactly where the pull of the points above and the points below is in balance: the signed residuals add to zero. It never chooses to hug the high points or the low ones — it splits the difference in the way that makes the total squared gap smallest.

3) It pivots through the centre. Say the average revision was \bar x = 5 hours and the average mark was \bar y = 5.2. Check: \hat y = 1.2 + 0.8 \times 5 = 5.2 = \bar y. It lands on \bar y exactly — no coincidence, as the next card shows.

It pivots through the means

The least-squares line is not free to sit anywhere: it always passes through the point of means (\bar x, \bar y). So if you knew nothing else, plugging the average x into the line returns the average y — the fit is "centred" on the data's centre of mass.

A regression line is only honest inside the range of the data it was built from. Push it beyond that and it can produce confident nonsense.

Suppose you fit a line to children's heights against age using data from ages 5 to 15. In that window it might be superb — predicting height within a centimetre. But children grow at roughly a constant rate over those years, so the line has a steady upward slope. Follow it out to age 40 and it cheerfully predicts an adult over 3 metres tall. The maths didn't break; growth simply stops, and the line has no way of knowing that because it never saw a grown-up.

Reaching far outside the observed range is called extrapolation, and it is one of the most common — and most expensive — mistakes in all of modelling. Predicting within the data is interpolation and is usually fine; predicting far beyond it is a leap of faith dressed up as arithmetic. Always ask: is the x I'm feeding in actually like the data the line was trained on?

The name is a lovely historical accident. In the 1880s the Victorian polymath Francis Galton plotted the heights of hundreds of grown children against the heights of their parents. He found the obvious thing — tall parents tend to have tall children — but also something subtler: the children of very tall parents were, on average, tall but slightly less tall than their parents; the children of very short parents were short but slightly less short. Heights seemed to drift back toward the average. Galton called this pull "regression toward mediocrity," and the technique of fitting the line inherited the word "regression" forever after.

The phenomenon itself — regression to the mean — is real and everywhere, and it fools people constantly. The team that tops the league one season tends to slip the next; the student who aces one test often scores closer to their average on the following one; the fund that was number one this year usually looks ordinary next year. Nothing needs to have caused the slip — extreme results are partly luck, and luck doesn't repeat. Yet people rush to credit or blame an intervention ("the new coach ruined them!") that in fact did nothing at all. Spotting regression to the mean is one of the great immunisations against being fooled by data.

See it explained