Overfitting and Underfitting

Two students walk out of the same exam looking equally miserable. Maya spent every evening for a month memorising last year's exact exam — every question, every model answer, word for word. Leo barely opened the textbook and picked up only the vaguest sense of the topic. This year's paper covers the same material, but every single question is worded differently.

Maya freezes. She recognises nothing — she learned specific answers, not the underlying ideas, so the moment the wording changes she has nothing to fall back on. Leo also fails, but for the opposite reason: he never grasped the underlying ideas in the first place, so he was doomed on any version of the exam, old or new. Two students, two opposite mistakes, the same result: failure on the exam that actually counts.

Machine learning models fail in exactly these two ways, and giving each one a name is the whole point of this page. It matters far beyond exams: a medical model that "underfits" a disease's real symptoms will miss cases it should catch, and a model that "overfits" the particular patients in its training hospital may perform brilliantly there and dangerously badly at a different hospital with a slightly different population. Getting this balance right is not a nice-to-have — it's the difference between a model that actually works in the world and one that only looks like it works, in the one place it was tested.

Underfitting — Leo's mistake. The model is too simple to capture the pattern in the data at all — like fitting a straight line through a trend that clearly curves. It does poorly on the training data and on new data, because it never learned the real pattern to begin with.
Overfitting — Maya's mistake. The model is too flexible, and instead of learning the general pattern it memorises the specific training examples, wiggles and noise included. It looks brilliant on training data and falls apart on anything new.
Just right. Complex enough to capture the real trend, simple enough to ignore the noise around it. Good on training data and good on new data — the actual goal.

Too simple, too wild, just right

The dots below follow a gentle curve with a little scatter mixed in — think of them as measurements with some natural noise. Step through three candidate models: a straight line (too simple), a wildly wiggly high-degree curve (too flexible), and a sensible curve in between. Watch what happens not just to the curve, but to the train error and test error printed above the chart — the train points are the ones the model was fitted to; the test points are new ones it has never seen.

The straight line misses the curve everywhere — its train error is already high, and its test error is nearly identical, because it was never capturing the shape of the data in the first place. The wild curve does the opposite trick: it bends itself to pass almost exactly through every training dot, so its train error looks fantastic — but between and beyond those dots it lurches wildly, so its test error is often worse than the straight line's. The curve in the middle gives up a little bit of training accuracy in exchange for a model that actually generalises — its train and test errors stay low and close together, which is the real signature of a good fit.

A second worked example: filtering spam

Suppose you're building a spam filter and you have 1,000 emails to train on, and a separate batch of 1,000 fresh emails to test on afterwards. Three attempts, three outcomes:

Attempt 1 — one rule. "If the email contains the word 'money', mark it spam." Dead simple, and it misses most real spam and flags plenty of ordinary emails too: 62% accuracy on training emails, 60% on test emails. Bad on both — this is underfitting. The rule is too crude to capture what actually makes an email spam.
Attempt 2 — a giant lookup table. Memorise the exact text of every training email and its label. On the training set this scores a perfect 100% accuracy — of course it does, it has literally seen every answer. On the fresh test emails, which it has never encountered, it scores only 55% accuracy — barely better than a coin flip. This is overfitting: a huge gap between training and test performance is the tell.
Attempt 3 — a learned pattern. A model that picks up on genuine spam signals — suspicious links, urgency words, sender reputation — scores 91% on training emails and 89% on test emails. Both numbers are good, and they're close together. That closeness is exactly what you want to see.

Notice the shape of the diagnosis in both examples (the curve-fitting one and the spam filter one): it is never just one number that tells you what went wrong. It's the relationship between the training performance and the test performance that reveals whether a model underfit, overfit, or got it just right.

There's a useful shortcut buried in both examples: look at the gap between training and test performance, not just the test number on its own. A small gap (attempt 3, or the "just right" curve) means the model is learning something that carries over to new data. A gap that stays huge no matter what you try (attempt 2, or the wild curve) is the fingerprint of a model with too much freedom for the amount of data it has been given. And if both numbers are disappointing and close together (attempt 1, or the straight line), more freedom — not less — is what the model actually needs.

The dial you must tune

Model complexity is the master dial of machine learning, and the goal is always the same: turn it to the sweet spot where test error is lowest — not training error, test error. Turn the dial too far towards "simple" and you underfit; too far towards "flexible" and you overfit. Both directions fail, just for opposite reasons, which is why you can't fix overfitting by making a model even more powerful, or fix underfitting by feeding it even more of the same limited features.

Two of the standard tools for finding — and holding — that sweet spot get their own pages: setting aside data you never train on so you can honestly measure where you stand is the train/validation/test split, and deliberately discouraging a model from getting too flexible is called regularization. The precise mathematical way to think about the whole balance is the bias–variance tradeoff.

One more nuance worth holding onto: the "right" position on the dial isn't fixed forever — it moves as your dataset grows. A model that overfits badly on a hundred training examples might be a perfectly reasonable choice once you have a hundred thousand, because there is now enough data to pin down all those extra parameters instead of letting them drift toward memorising noise. This is exactly why collecting more (representative) data is often the single most effective way to fight overfitting — sometimes more effective than any amount of clever tuning.

A beginner's most common mistake: staring proudly at a training error of nearly zero and concluding the model is great. Training error alone tells you nothing about overfitting. A model that has simply memorised its training set will also show a beautifully low training error — right up until you show it something new.

The only honest diagnosis compares training error to error on data the model has never seen — which is precisely why real projects always hold back a chunk of data purely for evaluation. A low training error with a much higher error on that held-out data is the single clearest overfitting warning sign there is.

The other habit worth building early: more complex models — more parameters, more flexibility, more capacity to bend — are overfitting-prone by default. That isn't a reason to always pick the simplest model available (an overly simple model just underfits instead), but it does mean that every extra ounce of flexibility you give a model has to be earned by enough data to justify it, or reined in with a technique like regularization.

Imagine a forecaster who builds their prediction by fitting a model to yesterday's exact minute-by-minute readings — every gust of wind, every passing cloud, every tiny wobble in pressure — and trusts that pattern completely. Tomorrow's weather is never a carbon copy of yesterday's, so a forecast that clings to yesterday's exact noise, rather than the broader seasonal trend, tends to be worse than a simpler forecast that just says "expect something close to the average for this time of year."

That's overfitting in the wild: chasing every wiggle in the data you happen to have, instead of the durable pattern underneath it. The forecaster who captures the general trend — even with less fine-grained detail — beats the one who perfectly explains yesterday and understands nothing about tomorrow.

Overfitting isn't only a toy-dataset problem — it has caused real, serious harm. Several face-recognition systems have been trained mostly on photographs of one narrow group of people. The system doesn't just learn "what makes a face" in general; it quietly overfits to details that were common across its training photos — particular lighting conditions, particular skin tones, particular facial structures — rather than learning features that generalise to every face.

The result: excellent accuracy on faces that resemble the training set, and a sharp, sometimes alarming drop in accuracy on faces that don't. This is exactly the overfitting pattern from earlier in this page — great on the data it trained on, much worse on data it didn't — except here "the data it trained on" means "the people it was shown," and the consequence isn't a bad exam grade, it's real people being misidentified. It's one of the reasons diverse, representative training data is now treated as a safety requirement, not an afterthought.

The fix mirrors the fix for every overfitting problem in this page: broaden what the model actually learns from. Just as the spam filter needed to learn genuine spam signals rather than memorise exact emails, a face-recognition system needs training photos that span the full range of lighting, ages, and backgrounds it will meet in the real world — otherwise "the training set" quietly becomes a much narrower, much less fair stand-in for "everyone."