One model is a single opinion. An ensemble is a committee — many models whose
combined verdict beats any member. You already met one committee in
The three families are bagging (train models in parallel and average them), boosting (train models in sequence, each repairing the last), and stacking (train a small extra model to learn how to combine the others). Parallel, sequential, or learned — that is the whole taxonomy.
The deepest split is between bagging, where the base models are grown independently and never see each other, and boosting, where each model is built specifically to fix what the previous ones got wrong. Bagging's models could be trained on a thousand machines at once; boosting's must wait in line.
Stacking sits on top of either: a first layer of diverse base models makes predictions, and a second-layer "meta-model" learns the best way to blend those predictions into a final answer.
Bagging — short for bootstrap aggregating — trains many copies of the same kind of model, each on a different random bootstrap sample (drawn with replacement) of the data, then averages their predictions (or takes a majority vote). Because each model sees a slightly different dataset, each overfits to slightly different quirks, and averaging cancels those quirks out.
Through the bias–variance lens, bagging is a variance-reduction machine. It works best on base models that are individually low-bias but high-variance — flexible learners that fit the signal well but wobble wildly from sample to sample. Deep decision trees are the textbook case, which is exactly why the random forest is a bagged ensemble of deep trees. Averaging leaves the (already low) bias untouched while crushing the variance:
Boosting flips the strategy. It trains models one at a time in sequence, and each new model is told to concentrate on the examples the current committee still gets wrong — by re-weighting those hard cases, or by fitting the leftover errors directly. The models are then added together, so the ensemble slowly bends itself toward the truth.
Boosting's natural base model is the opposite of bagging's: a weak learner that is
high-bias but low-variance, such as a shallow "stump" of a tree that is barely more than a
coin flip. On its own such a learner is hopeless, but a long sequence of them, each patching the
residual mistakes of the last, combines into a powerful predictor. So boosting is a
bias-reduction machine — it manufactures a low-bias committee out of high-bias
parts. Its most famous incarnation, gradient boosting, is the subject of
Bagging and boosting combine many copies of one kind of model with a fixed rule (average, or weighted sum). Stacking is greedier: it takes a handful of deliberately different models — say a random forest, a boosted-tree model, a logistic regression and a nearest-neighbour model — and trains a second-layer meta-model to learn the best way to weigh their predictions against each other.
The intuition is that different model families make different kinds of mistakes: where the forest stumbles, the logistic model may be confident, and vice versa. The meta-model learns to trust each base model exactly where it tends to be right. Because it exploits the diversity of genuinely different learners, a good stack can shave off both bias and variance — at the cost of real complexity and a serious risk of leakage if you build it carelessly (see the warning below).
| Bagging | Boosting | Stacking | |
|---|---|---|---|
| How models train | In parallel, independently | In sequence, each fixes the last | Base models in parallel, then a meta-model |
| Base model | Low-bias, high-variance (deep trees) | High-bias, low-variance (weak learners) | Several diverse model families |
| Mainly reduces | Variance | Bias | Both (via diversity) |
| Combined by | Averaging / voting | Weighted sum | A learned meta-model |
| Flagship | Random forest | Gradient-boosted trees | Competition-winning blends |
The classic stacking blunder is to train the base models on the training set, ask them to predict that same training set, and feed those predictions to the meta-model. But the base models have already seen those rows, so their in-sample predictions are unrealistically good — the meta-model learns to trust them far more than it should, and the whole stack collapses on real data. The fix is out-of-fold predictions: split the training data into folds, and for each row use only base models that were not trained on that row to generate its meta-features. This is the same leakage discipline as cross-validation, applied one level up. Skip it and your stack will look brilliant in development and fall apart in production.
In 2009 Netflix paid out $1 million to the team that best improved its movie-recommendation engine. The winning entry wasn't a single clever model — it was a gigantic stack, blending hundreds of different predictors into one. It worked, and it made ensembling famous, but there was a punchline: Netflix reportedly never deployed the winning blend, because the engineering cost of running that many stacked models in production outweighed the accuracy gain. It is a perfect cautionary tale — stacking can squeeze out the last drop of accuracy, but that last drop is often too expensive and fragile to ship. Bagging and boosting, being simpler, are far more common in real systems.
Kaggle Learn's Intermediate Machine Learning course builds ensembles on real competition data, where boosted trees are the default weapon and blending several models is a standard leaderboard trick. Practise there to feel which ensemble helps which problem.