Give a decision tree a training set and turn it loose with no limits, and it will happily keep asking questions — one more split, then one more, then one more — until every single training example sits alone in its own perfectly pure leaf. At that point the tree scores a flawless 100% accuracy on the data it was trained on. It has learned the training set perfectly.

That should sound like a red flag rather than a triumph, and it is. The tree hasn't found the real pattern connecting features to labels — it has essentially memorised the training set, quirks, noise, coincidences and all. Hand it a brand-new example that doesn't exactly match anything it saw during training, and it can perform badly, sometimes shockingly so. Trees are notoriously prone to this failure — more so than most other models, precisely because nothing stops them from growing until every last data point has its own private rule.

Why trees overfit so easily

Every split a tree makes only has to help carve out one subgroup a little better — it doesn't need to reflect any broad trend at all. Give the tree enough splits, and it can keep drilling down into smaller and smaller subgroups until it reaches a split that isolates a single noisy outlier point, purely because doing so raises that node's purity. Nothing in the greedy splitting process (choosing the cut with the highest information gain at each step) knows or cares whether the pattern it just found is a genuine trend or a one-off accident — it only ever asks "does this split make the current group purer?" A single weird data point is more than enough of an excuse for one more split.

This is really the same story told everywhere in machine learning — a model that's complex enough can always fit its training data better by fitting the noise, at the cost of performing worse on new data — trees simply have an unusually easy time doing it, because adding one more leaf to isolate one more troublesome point is always cheap and always available.

Deeper isn't better

Increase the depth slider below and watch the boundary bend itself around the noisy points, one kink at a time. The training accuracy (the coloured dots) climbs toward 100%, but the test accuracy (the squares — points the tree never saw while training) peaks at a modest depth and then falls as the tree starts contorting itself around noise instead of following the real boundary. That growing gap between training and test accuracy is overfitting, made visible.

Worked example: full tree vs. pruned tree

Imagine training a tree on 200 emails labelled "spam" or "not spam," then checking it against 50 fresh emails it never trained on.

Fully-grown tree. Left unchecked, it keeps splitting until every one of the 200 training emails lands in its own pure leaf: 100\% training accuracy. But several of those leaves exist only to handle one bizarrely-worded spam email or one oddly formal real email — quirks, not patterns. On the 50 held-out emails, it scores only 76\%.

Pruned tree. Cut back to a handful of sensible splits — "contains a link AND promises money," "sender not in contacts AND all-caps subject," and a few more — it gets a slightly worse training score, 91\%, because it no longer bothers chasing every last training email. But on the 50 fresh emails it scores 88\% — a large improvement, because those few splits captured genuine spam signals rather than one-off wording.

Look at what happened: lower training accuracy paired with higher test accuracy. That trade is almost always a sign of a healthier model, and it's exactly why you never judge a tree by its training score alone.

Two ways to hold a tree back

Both of the standard fixes work by refusing to let the tree chase every last training point:

• Limit the max depth. The simplest option: just don't let the tree grow past a chosen number of splits, however tempting a further split looks. It's cheap and easy, but picking the right number takes some trial and error.
• Pruning. Grow the full, over-eager tree first — then walk back through it and cut off branches that don't meaningfully improve accuracy on separate validation data (data set aside and never used for training). A branch survives only if removing it would actually make validation performance worse.

The sweet spot

The best depth (or the best pruned shape) is the one where test — or, during development, validation — accuracy peaks: complex enough to catch the real pattern, simple enough to ignore the noise. You find it by checking performance on held-out data, never on the training data itself. This is one instance of a universal theme, the bias–variance tradeoff — and one clever way to dodge it almost entirely is to grow many shallow-ish, individually-overfitting trees and average their answers together, the idea behind random forests.