Regression Metrics

A classifier's answers are right or wrong, so you count them. A regression model predicts a number — a house price, tomorrow's temperature, a patient's blood pressure — and it is essentially never exactly right. Predicting a house will sell for £312,000 when it goes for £315,000 is a fine prediction; predicting £120,000 is a disaster. So the question is not "how often is it right?" but "how wrong is it, and in what way?"

Every regression score is built from one raw ingredient — the residual, the gap between a true value y_i and the model's prediction \hat{y}_i:

e_i = y_i - \hat{y}_i.

The metrics on this page are all different ways of boiling a whole cloud of residuals down to one honest number. They disagree, on purpose, about how much a big miss should count — and choosing the right one is a modelling decision, not a formality.

MAE: average size of a miss

The mean absolute error is the plainest thing you could compute — the average distance between prediction and truth:

\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n}\bigl|\,y_i - \hat{y}_i\,\bigr|.

Its great virtues are that it is in the same units as the target (an MAE of 3,000 on house prices means "off by about £3,000 on a typical home") and that it is robust: every residual contributes in proportion to its size, so one freak error moves it only a little. Its one drawback is mathematical — the absolute value has a corner at zero, so MAE is not smoothly differentiable, which is why it is a fine report card but an awkward training objective.

MSE and RMSE: punishing the big misses

The mean squared error squares each residual before averaging:

\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^2.

Squaring changes the whole character of the score. A residual of 10 contributes 100; a residual of 1 contributes only 1. So MSE cares far more about a few large errors than about many small ones — it penalises outliers hard. It is also perfectly smooth and differentiable, which is exactly why it is the workhorse training loss for regression. This is the same MSE you met as the objective being minimised in the cost function; here we are simply reusing it to score a finished model rather than to train one.

MSE's blemish is its units: squared pounds, squared degrees — meaningless to a human. The fix is to take the square root, giving the root mean squared error:

\mathrm{RMSE} = \sqrt{\mathrm{MSE}} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^2}.

RMSE lives back in the target's units, like MAE, but it keeps MSE's temperament: because the squaring happens before the root, big residuals still dominate. A useful fact worth remembering is that \mathrm{RMSE} \ge \mathrm{MAE} always, and the gap between them is itself a signal — a wide gap means your errors are lopsided, with a few large ones dragging RMSE up above the typical miss.

Watch one outlier split MAE from RMSE

Here is the difference made visible. A handful of predictions have small, ordinary residuals, and one point is an outlier whose error you control with the slider. Watch what happens as you drag its error out: MAE creeps up in a straight line, while RMSE curves upward, accelerating away, because the outlier's squared error grows quadratically.

This is the whole choice in one picture. If a big miss really is much worse than a small one — a wildly mispriced house, a dangerously wrong dosage — you want RMSE's harsh punishment, so minimise it. If instead the occasional huge error is just noisy data you don't want dominating the score, MAE gives you a steadier, more representative "typical miss." Neither is more correct; they encode different beliefs about what a bad prediction costs.

R²: what fraction of the variation did we explain?

MAE and RMSE tell you the size of the error but not whether that size is good or bad — is an RMSE of 5 impressive? It depends entirely on the spread of the data. The coefficient of determination, R^2, fixes this by comparing your model against the laziest possible baseline: always predicting the mean \bar{y}.

R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2} = 1 - \frac{\mathrm{SS}_{\text{res}}}{\mathrm{SS}_{\text{tot}}}.

The denominator \mathrm{SS}_{\text{tot}} is how much the target varies on its own; the numerator \mathrm{SS}_{\text{res}} is how much variation your model failed to account for. So R^2 is the fraction of the variance your model explains, and it is unitless, which makes it easy to compare across problems:

R^2 = 1 — perfect: every residual is zero.
R^2 = 0 — your model is no better than blindly predicting the mean.
R^2 < 0 — worse than predicting the mean. Yes, it can go negative: nothing in the formula stops \mathrm{SS}_{\text{res}} from exceeding \mathrm{SS}_{\text{tot}}, and on unseen test data a bad model routinely does.

Adjusted R²: don't reward useless features

Plain R^2 has a mischievous property: adding any new feature, even pure random noise, can only push it up or leave it flat — never down. Give a model enough junk columns and its R^2 on the training set drifts toward a flattering 1 without the model actually being any good. The adjusted R^2 corrects for this by charging a fee for every feature p you spend, given n data points:

R^2_{\text{adj}} = 1 - \bigl(1 - R^2\bigr)\,\frac{n - 1}{n - p - 1}.

A new feature now only raises the adjusted score if it earns its keep — if it improves the fit by more than a random feature would be expected to by luck. This makes adjusted R^2 the fairer number when you are comparing models with different numbers of features, and a drop in adjusted R^2 after adding a variable is a clean signal that the variable is dead weight.

The five at a glance

Metric	Units	Outlier sensitivity	Reach for it when…
MAE	Same as target	Low (robust)	You want a plain "typical miss" and don't want rare huge errors to dominate.
MSE	Target squared	High	You are training — it is smooth and differentiable.
RMSE	Same as target	High	You want a readable error and big misses to be punished hard.
R²	Unitless	Medium	You want "how much better than predicting the mean?", comparable across problems.
Adjusted R²	Unitless	Medium	You are comparing models with different numbers of features.

Squaring looks arbitrary until you notice three things it buys you at once. First, it is smooth — a parabola has a derivative everywhere, including at zero, whereas the absolute value has a sharp corner there, so squared error slots straight into gradient-based optimisation while absolute error fights it. Second, it connects the metric to variance: minimising squared error is minimising a variance, which is why the mean-predicting baseline in R^2 falls out so naturally. Third, and most beautifully, minimising squared error is exactly the same as finding the most likely model if you assume the noise is Gaussian bell-curve noise — least squares is maximum-likelihood estimation in disguise. Gauss himself made this argument in 1809. Absolute error corresponds instead to a different noise assumption (a Laplace distribution) and estimates the median rather than the mean, which is precisely why it shrugs off outliers.

Two traps snare people constantly. First, R^2 = 0.8 does not mean "the model is 80% accurate" — regression predictions are never simply right or wrong. It means the model explains 80% of the variance in the target, which is a statement about spread, not a hit rate. Second, many people memorise "R^2 is between 0 and 1." That is only guaranteed on the training data for a model fitted by least squares. On a test set, or for any model that predicts worse than the mean, the residual sum of squares can exceed the total sum of squares and R^2 goes negative — sometimes wildly so. A negative test R^2 is not a bug; it is the metric correctly telling you that a flat horizontal line would have beaten your model.

Learn this on Kaggle

Kaggle Learn's Intermediate Machine Learning course has you score regression models on held-out data, where MAE is a one-line call and the leaderboard is ranked by RMSE. Practise there on a real housing dataset and feel how the choice of metric changes which model looks best.