Decision Trees

A doctor asks: "Is the patient's temperature over 38°C?" A loan officer asks: "Is income over $50,000?" A friend playing 20 Questions asks: "Is it bigger than a breadbox?" Each of them is doing the exact same thing — narrowing down an answer with a branching series of simple yes/no questions, each one chosen because it splits the remaining possibilities in the most useful way.

A decision tree turns that everyday habit into a model. It predicts by asking a sequence of simple questions, like a flowchart: "Is feature 1 above 2? If yes, is feature 2 below 0?" Each question splits the data, and you follow the branches down to a leaf that gives the prediction. It is one of the most human-readable models in machine learning — you can trace the exact chain of reasoning behind any single prediction, question by question, the same way you'd trace your own thinking out loud.

Anatomy of a tree

Every decision tree is built from three kinds of parts, and the vocabulary matters because you'll meet it everywhere trees are discussed:

The root node — the very first question, sitting at the top. It's chosen to be the single most useful question you could ask about the whole dataset.
Internal nodes — every follow-up question further down the tree, each one only seeing the slice of data that survived the questions above it.
Leaf nodes — the final answer. Once you reach a leaf, there are no more questions; it just hands back a prediction.

How does a tree decide which question is "most useful" at each node? It measures how mixed-up a group of examples currently is, and how much a candidate question would clean that mixture up — that's exactly what entropy and information gain measure. At every node, the tree tries every candidate split, scores each one, and greedily asks whichever question gains the most. Do that node after node, and a whole tree grows itself automatically, one locally-best question at a time.

Worked example: following a tree to a prediction

Suppose a tiny weather tree has already been built from past data, to predict "will it rain this afternoon?" using three features — the outlook, the humidity, and the wind:

Root: "What's the outlook?"
- Overcast → Leaf: Rain. (Overcast skies here bring rain the vast majority of the time — no more questions needed.)
- Sunny → ask the next question, below.
- Already raining → ask a different next question, about the wind.
Internal node (under "Sunny"): "Is humidity high?"
- High → Leaf: Rain. (Humid, sunny afternoons here often build into a shower.)
- Normal → Leaf: No rain.
Internal node (under "Already raining"): "Is the wind strong?"
- Strong → Leaf: No rain. (Strong wind blows the rain clouds through quickly.)
- Weak → Leaf: Rain. (Calm rainy days tend to linger.)

Now use it on a brand-new day: outlook = Sunny, humidity = High, wind = Weak. Follow the path — start at the root and ask "what's the outlook?" It's Sunny, so move to the internal node underneath and ask "is humidity high?" It's High, so you land on a leaf: the tree predicts Rain. Notice the wind reading was never even asked about — once you reach a leaf, the tree is done, no matter how many other features exist.

Worked example: choosing the very first split

Where does a question like "what's the outlook?" come from in the first place? It's chosen by comparing the information gain of every candidate question and picking the biggest one — exactly the calculation from entropy and information gain. Recall the 8-customer group from that page (entropy 0.811 bits, since 6 of the 8 bought something) and its two candidate splits:

"Age over 30?" — information gain 0.311 bits.
"Owns a pet?" — information gain 0.204 bits.

A tree being built on this data compares every candidate question the same way, then greedily picks the biggest number. Since 0.311 > 0.204, "Age over 30?" wins the root node — it's the single most useful question to ask first, because it cleans up more of the group's uncertainty than any other option on the table. The tree then repeats the exact same contest separately inside each of the two children it just created, growing downward one greedy choice at a time until it runs out of useful questions to ask.

Carve up the space

Zoom out from any one dataset, and a trained tree's questions carve the whole feature space into rectangular boxes, each labelled with a class — because every question is a single axis-aligned cut, a straight slice across just one feature at a time. Slide the two cuts below: a vertical split on feature 1 and a horizontal split on feature 2. They divide the plane into four boxes, each coloured by the majority class of the points inside it. Find cuts that put each class neatly in its own box — that's what training a tree does, automatically, question after question.

Why trees are loved

Decision trees are interpretable — you can read the rules straight off the flowchart, out loud, to someone with no maths background at all — and they happily mix numbers with categories, need no feature scaling, and capture non-linear, boxy boundaries a single straight line never could.

Splits are always one feature, one threshold, at a time. A tree can never draw a diagonal boundary in one step — unlike logistic regression's single straight line, or a split that weighs several features together at once. If the true pattern in the data runs diagonally across two features, a tree can only approximate it with an awkward staircase of tiny boxy steps, never a clean diagonal cut.
An unconstrained tree overfits notoriously easily. Left to keep splitting until every leaf is perfectly pure, a tree will happily carve out one tiny box per training example — memorising the training data instead of learning a pattern that generalizes to new examples. Next up: why that happens, and how to stop it.

Play 20 Questions and you're building a decision tree live, one question at a time, entirely by intuition. "Is it alive?" splits the whole universe of possible answers roughly in half — a great root node. "Is it bigger than a car?" a few turns later is a much narrower, more specific split — a deep internal node, only useful once earlier questions have narrowed the field. A skilled player is unconsciously doing something close to maximising information gain at every turn: always reaching for the question that eliminates the most possibilities, exactly like a tree greedily choosing its next split.

Real medical triage flowcharts, credit-approval systems, and customer-service chatbots lean on decision trees (or small forests of them) precisely because a human can trace through and say exactly why a specific decision was made — "you were declined because income was under $50k and existing debt was above a threshold" is a sentence a regulator, a doctor, or a customer can actually check. A neural network arriving at the same decision offers no such sentence — just a pile of numbers multiplied together, with no step you can point to and say "that's why." When being able to explain a decision matters as much as getting it right, trees are often the model of choice for exactly that reason.