The Bias–Variance Tradeoff

Picture four archery clubs, each firing a round of arrows at the same target.

Club A's arrows all land close together, but consistently off to the upper-left — tight, but wrong in the same way every time.
Club B's arrows are scattered all over the target — sometimes bullseye, sometimes miles off — but if you averaged every arrow's position, that average would land near the centre.
Club C's arrows land tightly clustered right on the bullseye, round after round.
Club D's arrows are both scattered and consistently off to one side — the worst of both worlds.

Club A is biased but consistent — like an underfitting model that is too simple to hit the true pattern, but at least gives the same (wrong) answer every time. Club B is unbiased on average but wildly inconsistent — like an overfitting model that nails the training data but swings unpredictably on anything new. Club C is what every model wants to be. Club D is what nobody wants. A model's error on new data always splits into exactly these two ingredients — bias and variance — and, as you are about to see, improving one has an uncomfortable habit of worsening the other.

This page takes the informal underfitting/overfitting story from the previous page and gives it precise mathematical footing. "Too simple" and "too flexible" are useful words for a beginner, but "high bias" and "high variance" are the words you'll find in every research paper, every job interview, and every serious discussion of why a model behaves the way it does — because they name the two error sources separately, and separate names let you diagnose which one you're actually fighting.

Bias and variance, defined

Bias — the systematic error that comes from a model being too simple, too rigid, to represent the true pattern. A biased model is consistently wrong in the same direction, no matter which training data you hand it. This is the mathematical fingerprint of underfitting.
Variance — the error that comes from a model being oversensitive: retrain it on a slightly different sample of the same kind of data, and its predictions swing wildly. A high-variance model isn't wrong in a consistent direction — it's just unreliable. This is the mathematical fingerprint of overfitting.

Both are measured the same way: imagine training your model many times, each time on a different random sample drawn from the same population. Bias asks "how far off, on average, do these trained models land from the truth?" Variance asks "how much do these trained models disagree with each other?" A great model scores low on both questions.

A numeric worked example

Suppose the true relationship you're trying to learn always outputs 50 for a certain input, and you train five times on five different (but similar) samples of data.

A very simple model (say, "always predict the overall average") produces: 31, 32, 30, 31, 32. Barely any spread between runs — that's low variance — but every single prediction is stuck about 19 points below the truth. That consistent gap is high bias. This is a straight line trying to fit a curve: stable, but stably wrong.
A very complex model (say, a high-degree curve with dozens of tunable knobs) produces: 22, 71, 38, 65, 49 across the same five training runs. The average of those five numbers is close to 50 — so its bias is nearly zero — but no individual run is anywhere near reliable. That enormous run-to-run spread is high variance. This is the wild, overfit curve from the earlier page: it can match the truth on average, yet you can't trust any single trained copy of it.

Neither model is genuinely good. The fix isn't to swing to the opposite extreme — it's to find a model complex enough to lower that first model's bias, without picking up the second model's variance along the way.

It helps to picture what "training five times" really represents. In practice you rarely train the exact same model five separate times on purpose — but every time you collect a new batch of data (a new group of patients, a new week of sensor readings, a new class of students), you are effectively sampling from the same underlying population all over again. Bias and variance describe how a fixed modelling choice — not any one specific trained model — would behave across that whole range of possible samples, which is precisely why they're properties of a method, not of a single trained result.

The U-shaped curve

Watch the three curves as complexity grows from left to right. Bias² starts high and slides down — more complexity means more room to fit the true pattern. Variance starts low and climbs — more complexity means more room to fit noise too. Their sum, the total error, dips to a minimum somewhere in the middle and then rises again. Drag the complexity marker until the total-error dot sits at the very bottom of that U — that complexity, not the smallest or the largest available, is the one you actually want.

Notice you can't push the marker left and right at the same time. Sliding towards "simple" always trades some variance away for more bias; sliding towards "complex" always trades some bias away for more variance. There is no setting that minimises both at once — that tension is the tradeoff the name promises.

Steering the tradeoff

Every knob in machine learning is, secretly, a bias–variance knob: a tree's depth, a polynomial's degree, the strength of regularization, the size of a network. Turn any of these towards "more flexible" and you push bias down but variance up; turn towards "more constrained" and it's the reverse. More data is the rare free lunch — it lowers variance without raising bias, shifting the whole sweet spot toward more powerful models, which is why the same model architecture can underfit on a small dataset and comfortably generalise once given a much larger one. Knowing which way a change pushes you is half of practical machine learning.

This is also why "just add more parameters" and "just simplify the model" are both, on their own, bad advice — the right move always depends on which side of the U you're currently standing on. A model with high training error and high test error close together needs more capacity or better features (it's stuck on the high-bias side); a model with near-zero training error and much higher test error needs more data, regularization, or fewer parameters (it's stuck on the high-variance side). Reading off the diagnosis correctly, before reaching for a fix, is the skill this whole page is really about.

It's tempting to think of bias and variance as two separate bugs you can each squash for good. You can't. You cannot minimise bias and variance independently and without limit — for a fixed amount of training data, pushing model complexity down to reduce variance necessarily raises bias, and pushing it up to reduce bias necessarily raises variance. This isn't a limitation of today's algorithms waiting to be engineered away; it's a mathematical fact about splitting one number (total expected error) into two competing pieces.

The full picture has a third piece too:

\text{expected error} = \text{bias}^2 + \text{variance} + \text{irreducible noise}

That last term, irreducible noise, is randomness genuinely present in real data — measurement error, unpredictable human behaviour, factors nobody recorded. No model, however cleverly tuned, can ever drive total error below that noise floor. The practical goal is never "zero error" — it's getting bias² and variance as small as the data honestly allows, and accepting the rest.

It's worth sitting with why bias is squared in that formula and variance isn't. Bias measures a signed average gap between a model's typical prediction and the truth — squaring it turns that gap into a genuine error contribution regardless of whether the model tends to overshoot or undershoot. Variance, by its own definition, is already a spread (an average squared deviation), so it needs no further squaring. The two terms are built from slightly different raw ingredients, but both land in the same units — error — which is exactly what lets you add them together honestly.

Two forecasters, two failure modes. The first always predicts "the seasonal average for today's date, no matter what" — reliable and consistent, but consistently a bit off on any day that isn't perfectly average. Zero variance, but real bias.

The second forecaster throws out the calendar and extrapolates wildly from this morning's single reading — "the temperature is rising fast right now, so tomorrow must be a scorcher." Their forecasts might average out about right over a whole year, so their bias is low, but any individual forecast swings unpredictably and is often badly wrong. High variance, low bias.

A genuinely good forecaster does neither — they blend the long-run seasonal pattern with today's actual conditions, which is precisely why random forests exist: instead of trusting one wildly-swinging decision tree (low bias, high variance), a random forest trains many different trees on different random samples and averages their predictions. Averaging cancels out a lot of the swinging without reintroducing the bias a single simple model would have — variance goes down, bias stays low. It's the machine-learning equivalent of asking a hundred forecasters and taking the average, rather than betting everything on just one.