Random Forests

Suppose you need a diagnosis and you're only allowed to ask one doctor. Ask a brilliant doctor and you're fine — but ask one who is having an off day, or one who has only ever seen patients quite unlike this one, and you get a confident, wrong answer, with no way to know it's wrong. Now imagine asking a hundred different doctors, each trained slightly differently on a slightly different set of cases, and going with whatever most of them agree on. A single doctor's blind spot rarely survives being outvoted by ninety-nine others who don't share it.

That is exactly the trick behind a random forest. A single decision tree is a brilliant but overconfident doctor: grown deep enough, it will happily memorise the quirks of its training data and overfit. So instead of trusting one tree, a random forest grows hundreds of them, each one trained on a slightly different, randomly-scrambled slice of the data, and lets them vote. It is quite literally the "wisdom of crowds" turned into mathematics: each tree's individual mistakes are idiosyncratic and point in different directions, so averaging them cancels the mistakes out while the real signal — which every tree can see, however dimly — survives the vote.

Two deliberate doses of randomness

For a vote to help, the trees can't all be near-identical copies of each other — a hundred votes for the very same overfitted answer is still just that one wrong answer, only louder. A random forest manufactures genuine diversity between its trees in two separate ways, and the word "random" in its name refers to both of them at once.

Bagging (bootstrap aggregating). Each tree is not shown the whole training set. Instead it is handed a random sample of the same size, drawn with replacement — so some training examples get picked two or three times for a given tree, and others get left out entirely, purely by chance. Every tree therefore studies a slightly different textbook. That matters because it means each tree ends up overfitting to a slightly different set of quirks — and quirks that differ from tree to tree are exactly the kind of error that averaging can cancel out.
Random feature selection. When a tree looks for its next best split, it isn't allowed to consider every available feature — only a small random subset of them. This stops every tree from greedily latching onto the single strongest feature first and building a near copy of every other tree around it. Forcing some trees to make do without the "obvious" feature pushes them to discover other, weaker patterns instead — which decorrelates the trees, and decorrelated errors are precisely what a vote is good at cancelling.

Both tricks are aiming at the same target from different angles: diversity. A forest of a thousand trees that all happen to make the same mistake is no better than one tree that makes it — the whole scheme only works if the mistakes disagree with each other. Sketched as a loop, growing a forest looks like this:

interface Row { features: number[]; label: number } function trainForest(data: Row[], numTrees: number, featuresPerSplit: number): Tree[] { const forest: Tree[] = []; for (let i = 0; i < numTrees; i++) { const sample = sampleWithReplacement(data, data.length); // bagging const tree = growTree(sample, featuresPerSplit); // random feature subset at each split forest.push(tree); } return forest; } function predict(forest: Tree[], row: Row): number { const votes = forest.map((tree) => tree.predict(row)); return majorityVote(votes); // or: average the votes for a probability }

Every tree runs the ordinary tree-growing algorithm you already know — it's only the inputs to that algorithm, the sampled rows and the shrunken menu of features, that are randomised. Nothing about how a single tree decides on a split changes at all.

Jagged trees, smooth forest

You can see both sources of randomness pay off directly in the shape of the decision boundary. Toggle between a single tree's boundary and the forest's below. The lone tree's boundary is jagged, twitching around individual points it happened to memorise. The forest's boundary — literally the average of hundreds of such jagged trees, each jagged in its own random way — is smooth and sensible, hugging the true underlying trend instead of the noise.

Worked example: outvoting the mistakes

Imagine five trees in a (tiny, toy) forest have each been asked whether a loan application should be approved or denied, based on a new applicant they've never seen. Suppose the honest, correct answer is deny. The five trees, each trained on its own random sample of past applicants and its own random subset of features, come back with:

Tree 1: Deny ✓
Tree 2: Approve ✗
Tree 3: Deny ✓
Tree 4: Deny ✓
Tree 5: Approve ✗

Two of the five trees — a substantial 40% of them — got this applicant wrong. But the vote is 3 to 2 in favour of "deny," so the forest's final answer is correct, even though it was built entirely out of individually fallible parts. A forest doesn't need every tree to be right; it only needs the trees' mistakes to disagree with each other more often than they agree. (A real forest usually reports the share of trees that voted each way — here, "60% confidence in deny" — rather than only the raw winner, which is often more useful than a bare yes or no.)

Written as a formula, with T trees each returning a vote v_i \in \{0, 1\} for "deny", the forest's estimated probability of "deny" is simply their average:

P(\text{deny}) = \frac{1}{T}\sum_{i=1}^{T} v_i

Here that's \frac{3}{5} = 0.6. This "soft voting" by average is usually preferred over a bare majority winner, because 3-out-of-5 and 99-out-of-100 both round to the same hard "deny," yet they represent very different amounts of underlying agreement.

Worked example: keeping the bias, killing the variance

There's a sharper way to say why this works, using the bias–variance tradeoff. A single, fairly deep decision tree typically has low bias — it's flexible enough to bend around almost any real pattern in the training data — but high variance: retrain it on a slightly different sample and it can carve up the feature space quite differently, because it's so sensitive to exactly which points it happened to see.

Averaging doesn't touch the bias — the average of many low-bias trees is still low-bias, since each one is (on average, across all the different random samples they might have seen) aiming at the right target. But averaging crushes the variance. If the trees' errors were completely independent of one another, the variance of the average of n trees would shrink all the way down to:

\text{Var}(\text{forest}) = \frac{\text{Var}(\text{one tree})}{n}

In practice the trees aren't fully independent — they're all trained on overlapping data — so the real reduction is smaller than a clean 1/n. But bagging and random feature selection exist precisely to push the trees as far towards independence as practically possible, so that this formula comes as close to true as it can: keep the low bias of a deep tree, and trade away as much of its variance as the randomness can buy.

Growing a forest instead of one tree isn't a free lunch — it comes with two real costs:

You lose the easy story. A single decision tree can be read aloud as one clean chain of if-then rules — "income above X and existing debt below Y, so approve." A random forest is the average of hundreds of such trees, so there is no longer one path to point to: you can rank which features mattered overall across the forest, but you can't trace one simple line of reasoning behind any single prediction. That's a genuine cost wherever an explanation is part of the job — a bank justifying a declined loan, a doctor justifying a diagnosis — which is one reason logistic regression, with its plain, additive formula, still gets chosen over a more accurate forest when explainability is what actually matters.
More trees help — but less and less, and never for free. The first fifty trees knock a big chunk off the variance; the next fifty help a little; by a few hundred trees, adding more barely moves accuracy at all, because you've already averaged away almost all the variance there was to remove. Meanwhile every extra tree still costs real time and memory to train and to store, and makes every future prediction slightly slower to compute. It is not "infinite trees, infinite accuracy" — past a point you're only burning compute for a rounding error.

The same principle behind a random forest shows up far outside computer science. On the game show Who Wants to Be a Millionaire?, a contestant stuck on a question could poll the studio audience or phone one expert friend. Studied over many episodes, the audience poll — hundreds of ordinary, individually fallible people voting — was right roughly 90% of the time, comfortably beating the single expert "phone a friend," who typically landed around 65%. No one audience member needs to be reliable; their mistakes just need to point in different directions so the crowd's vote washes them out. It's the identical reasoning that lets prediction markets forecast elections surprisingly well, and it's a big part of why random forests remain one of the most dependable, "just works" algorithms in machine learning — reliably strong across a huge range of real business and science problems, usually with far less careful tuning than a neural network needs.

The wisdom of crowds, made mathematical

This general trick — combine many high-variance models to get one low-variance model — is called ensembling, and it shows up everywhere in machine learning, not just with trees. Random forests need little tuning, resist overfitting, and can report which features mattered, which is why they're such a common first choice on tabular data. (Their cousin gradient boosting takes the ensembling idea in a different direction — it builds trees one at a time in sequence, each one specifically trying to fix the previous trees' mistakes, and wins an enormous number of real-world competitions.) Ensembles close out the tree family of methods; next we step back and study fitting and evaluation in general.

One more thing a forest hands you almost for free: because it tracks how much each random feature subset actually helped across hundreds of trees, it can rank every feature by how useful it tends to be — a rough, practical answer to "which of these measurements actually matters?" even though no single tree's path can fully explain any one prediction. That combination — strong, low-fuss accuracy on almost any tabular dataset, plus a usable feature ranking — is exactly why random forests turn up everywhere from credit-risk scoring to species classification from sensor readings, long before anyone reaches for something as heavy as a neural network.