Gradient-Boosted Trees

If you had to bet on one algorithm to win a tabular-data competition, you would bet on gradient-boosted trees. They are the workhorse behind an enormous share of Kaggle victories and production models — the thing people reach for when the data is rows and columns rather than pixels or text. They are also the most elegant idea in the whole ensemble family, because they turn boosting into something you already understand: gradient descent, but performed over functions instead of numbers.

Boosting, recall, builds a committee sequentially, each new member repairing the mistakes of the ones before it. Gradient boosting makes "repair the mistakes" precise: each new tree is fitted to point in the direction that most reduces the loss — the negative gradient of the loss. Adding that tree is one step downhill.

The additive model

A gradient-boosted model is a sum of small trees, grown one at a time. Write F_m for the ensemble after m trees. We start with a trivial guess F_0 (often just the mean of the target) and then repeatedly bolt on one more tree:

F_m(x) = F_{m-1}(x) + \nu\, h_m(x).

Here h_m is the new shallow tree and \nu (nu) is the learning rate, a small number like 0.1 that shrinks each tree's contribution so the ensemble creeps toward the answer rather than lurching. The whole model is just this running total — a base guess plus a long series of small, scaled corrections.

Why "gradient"? Descent in function space

The magic is what each new tree is trained to predict. In ordinary gradient descent you nudge a parameter vector in the direction of the negative gradient of the loss. Gradient boosting does the same thing, except the object being improved is not a vector of numbers — it is the whole prediction function F. At each stage it computes, for every training point, the negative gradient of the loss with respect to the current prediction:

r_{i,m} = -\left.\frac{\partial L\big(y_i, F(x_i)\big)}{\partial F(x_i)}\right|_{F = F_{m-1}}.

These r_{i,m} are called pseudo-residuals, and the new tree h_m is trained by ordinary regression to predict them. There is a lovely special case that makes the whole scheme click: if the loss is squared error L = \tfrac{1}{2}(y - F)^2, then its negative gradient is exactly y - F — the plain residual, the leftover error. So in the most common case, each tree is literally fit to the mistakes of the ensemble so far, and the general gradient formula just extends that idea to other losses (logistic loss for classification, absolute-error loss for robustness, and so on). Boosting is gradient descent, one tree per step, in the space of functions.

Watch it converge — and then overfit

Because each tree only chips away a little, the training error falls steadily as you add more trees. But there is a catch that defines how you tune these models: past some point the ensemble starts fitting noise, and error on unseen data turns around and climbs. Slide the learning rate and watch both curves: a small rate descends slowly but overfits late; a large rate plunges fast but turns upward sooner.

This picture is the reason the two most important knobs — the number of trees and the learning rate — must be tuned together. A smaller learning rate almost always generalises better, but it needs more trees to get there, so you trade compute for accuracy. The usual recipe is: pick a small learning rate you can afford, then add trees until the validation error stops improving (a tactic called early stopping).

The knobs that matter

Hyperparameter	What it controls	Trade-off
n_estimators	Number of trees (boosting rounds)	More trees fit better but eventually overfit and cost compute.
learning_rate (ν)	How much each tree contributes	Smaller generalises better but needs more trees.
max_depth	How complex each individual tree is	Shallow trees (weak learners) are the point; deep ones overfit fast.
subsample	Fraction of rows/features per tree	Sampling less than 100% adds randomness that fights overfitting ("stochastic" gradient boosting).

Notice that gradient boosting keeps its trees deliberately shallow — often just a few levels deep. That is the opposite of a random forest, which grows deep trees and averages them. The reason is the bias–variance division of labour: a forest bags low-bias, high-variance deep trees to cut variance, while boosting sequences high-bias, low-variance shallow stumps to cut bias. Same raw material, opposite strategy.

The idea above is the whole story mathematically, but three libraries turned it into the dominant tool on tabular data, each with an engineering trick worth knowing (no code required):

XGBoost added strong regularization — an explicit penalty on tree complexity baked into the objective — plus a second-order (Newton-style) approximation of the loss, which finds better splits than the plain gradient alone.
LightGBM made it fast with histogram splits: instead of trying every possible threshold, it buckets each feature into a few hundred bins, so finding the best split is a quick histogram scan. It also grows trees leaf-wise rather than level-wise.
CatBoost specialises in categorical features, encoding them in an ordered way that avoids a subtle leakage that naive target-encoding causes.

All three are the same gradient-boosting algorithm underneath — the differences are speed, regularization and how they handle awkward data, not the core maths.

With a random forest, adding more trees never hurts — the extra trees just refine an average, and accuracy plateaus. Gradient boosting is the opposite: because each tree deliberately reduces bias by fitting the current residuals, running for too many rounds makes the ensemble start fitting the noise, and validation error rises. There is a genuine "too many trees" here. This is why you tune n_estimators against validation data and use early stopping — and why you must never pick the number of trees by looking at training error, which only ever falls. Boosting rewards patience with a small learning rate and punishes greed with overfitting.

Learn this on Kaggle

Kaggle Learn's Intermediate Machine Learning course has a whole lesson on XGBoost, where you tune the learning rate, the number of estimators and early stopping on a real dataset. It is the fastest way to feel the trees-versus-learning-rate trade-off in your hands.