Imagine a student who prepares for an exam by memorising the exact answers to last year's paper — word for word, number for number. Sit them down in front of this year's paper, with the questions shuffled and the numbers changed even slightly, and they're lost. They never learned the subject; they learned one specific piece of paper.
A machine learning model can make exactly the same mistake. The entire goal of training is generalization: learning the underlying pattern well enough to handle brand-new examples it has never seen, not just repeating back the examples it trained on. A model that nails its training data but flops on new data hasn't learned anything useful — it merely memorized.
Here's the distinction that this whole idea rests on: a model is trained on one batch of examples, but it's only useful if it works on data it did not see during training — tomorrow's photos, next week's customers, a sentence nobody has typed before. Checking a model only against the data it already trained on tells you almost nothing, for the same reason that grading the student on last year's paper tells you nothing about whether they understand the subject.
That's why, later on, you'll learn to keep some data locked away and hidden from training entirely,
just to test generalization honestly — the idea behind
It helps to say precisely what "new" means here. It doesn't have to mean data from next year or
from the other side of the planet — it just means examples the model's internal numbers were never
adjusted using. A photo the model has literally never had access to during
Both curves below pass close to the nine training dots. The wild overfit curve threads every single one of them exactly — but lurches wildly in between and badly misses the held-out test squares. The simple straight fit ignores the noise and predicts the test squares well. Toggle between the two and watch the test error tell the real story.
Put rough numbers on it, and the contrast is stark:
| Model | Error on training data | Error on brand-new test data |
|---|---|---|
| Overfit (memorized) | almost 0 — nearly perfect | large — badly wrong |
| Simple (generalized) | small, but not zero | small — still accurate |
The overfit model's dazzling training score is a trap. It looks like the better student right up until the new questions arrive — at which point the simple model, which never bothered chasing every last training dot, comes out far ahead.
A child who memorises "7 × 8 = 56" by rote, as one fact among a fixed list, can answer that
exact question instantly — but ask them
It's tempting to see a model score 100% on its training data and assume the job is done. Resist that temptation. A perfect training score is genuinely ambiguous — it can mean the model has truly captured the pattern, or it can mean the model has simply memorized every answer, the way our exam-cramming student did.
The only way to tell those two apart is to check performance on data the model has never seen during training. Training-data performance and real-world usefulness are not the same measurement — a lesson that leads directly to keeping a separate validation and test set, which you'll meet soon.
Here's what that ambiguity looks like across a real training run, watching both scores epoch by epoch:
| Epoch | Training error | Unseen (validation) error |
|---|---|---|
| 1 | 0.40 | 0.42 |
| 5 | 0.18 | 0.20 |
| 10 | 0.09 | 0.11 |
| 20 | 0.03 | 0.10 |
| 40 | 0.01 | 0.19 |
Up to around epoch 10, both scores fall together — genuine learning. Past epoch 10, the training error keeps sinking toward zero, looking better and better... while the validation error quietly turns around and starts climbing back up. That fork in the road, where the two lines stop agreeing, is the exact moment memorisation took over from generalization. Watching for that fork is precisely why the validation score, not the training score, is the one worth trusting.
A famous (and often retold, if hazily documented) cautionary tale from early machine learning research: a team trained a model to spot camouflaged tanks hidden in forest photographs. On their test photos, it worked brilliantly — until it mysteriously failed on new pictures. The investigation found the twist: every one of the training photos with a tank happened to have been taken on a cloudy day, and every photo without a tank was taken on a sunny day. The model had never learned to recognise tanks at all — it had generalized perfectly to a pattern that was actually just "cloudy sky versus blue sky," which was utterly useless the moment the lighting changed. It's the canonical reminder that a model learns whatever pattern is really in the data, whether or not that's the pattern you meant to teach it.
Point at three very different dogs — a wobbling chihuahua, a shaggy sheepdog, a sleek greyhound — and say "dog" each time, and most two-year-olds will confidently and correctly call the next hundred dogs they meet "dog," including breeds they've never seen. Many machine learning models, by contrast, need thousands or millions of labelled photos to learn the same category reliably. Human brains are extraordinarily good at generalizing from a handful of examples — a gap that an entire area of machine learning research, few-shot learning, is devoted to closing.
Good generalization is a balancing act: complex enough to capture the true pattern, simple enough
to ignore the noise. That tension has a name —