Hyperparameter Tuning

A gradient-boosted model has a learning rate, a tree depth, a number of rounds, a subsample fraction. A random forest has a tree count and a features-per-split. A neural network has a learning rate, a width, a depth, a dropout rate. None of these are learned from the data — you have to choose them before training even begins, and the right choice can be the difference between a mediocre model and a winning one. Finding good values is hyperparameter tuning, and it is a search problem in its own right.

First, a crucial distinction that trips people up.

Parameters are learned from the data during training — the weights of a regression, the split thresholds inside a tree. The training algorithm sets them.
Hyperparameters are set before training and govern how the learning happens — the learning rate, the tree depth, the regularization strength, the k in k-nearest-neighbours. You set them.

Training finds the parameters; tuning finds the hyperparameters. This page is about the second job.

Grid search: try every combination

The obvious approach is grid search: list a few candidate values for each hyperparameter, then train and score a model for every combination. Neat, exhaustive, embarrassingly parallel — and cursed. The number of combinations is the product of the list lengths, so it explodes:

\text{trials} = v_1 \times v_2 \times \cdots \times v_d.

Five values across six hyperparameters is 5^6 = 15{,}625 models to train. This is the curse of dimensionality in the tuning grid: each extra hyperparameter multiplies the cost. Worse, grid search spends that huge budget testing only a handful of distinct values per axis — exactly the flaw the next method exposes.

Random search: throw darts instead

Random search samples each hyperparameter from a range at random, for some fixed budget of trials. In 2012 Bergstra and Bengio showed the surprising result that, for the same number of trials, random search usually beats grid search. The reason is beautiful and purely geometric. Most hyperparameters barely matter; a few matter a lot. A grid wastes its budget re-testing the unimportant ones, so it only ever tries a few distinct values of the important one. Random search, by contrast, gives almost every trial a fresh value of every dimension.

Toggle between the two layouts below. Both spend nine trials. Look at the tick marks projected onto each axis: the grid probes only three distinct values per axis, while random search probes nine. If the horizontal axis is the one that matters, random search has learned about it in far more detail — for free.

Bayesian optimization: learn where to look next

Grid and random search are blind — every trial is chosen without looking at how the previous ones turned out. Bayesian optimization is smarter: it treats the unknown "hyperparameters → validation score" relationship as a function to be learned, and uses each result to decide where to sample next.

A surrogate model (often a Gaussian process) is fitted to the trials so far. It predicts the score at any untried setting and how uncertain that prediction is.
An acquisition function (such as expected improvement) then scores every candidate setting by how promising it is — balancing exploitation (sample where the surrogate predicts a high score) against exploration (sample where it is most uncertain). The next trial is wherever the acquisition function is highest.

Because it concentrates its trials on promising regions instead of scattering them blindly, Bayesian optimization typically reaches a good model in far fewer trials — invaluable when a single trial means hours of training. The price is that the trials are now sequential (each needs the last one's result), so it parallelises less naturally than grid or random search.

The three strategies compared

Method	How it picks trials	Strength	Weakness
Grid search	Every combination on a fixed grid	Simple, fully parallel, reproducible	Cost explodes with dimensions; wastes budget on unimportant axes
Random search	Random samples from ranges	Better coverage per trial when few dims matter	Still blind to past results
Bayesian optimization	A surrogate + acquisition function chooses the next point	Reaches good settings in far fewer trials	Sequential, more complex to run

Always tune against validation data — never the test set

Whichever search you use, each candidate setting must be scored honestly, and the gold standard is cross-validation: for every hyperparameter combination, run k-fold CV on the training data and average the validation scores. The setting with the best cross-validated score wins.

The one iron rule is that the test set never enters the tuning loop. The moment you pick hyperparameters by peeking at test performance, the test set has helped build the model, so it can no longer give an honest estimate of how the model does on truly unseen data — you will have quietly overfit to it. Tune on validation folds; unlock the test set exactly once, at the very end, to report the final number.

It feels wrong that random sampling could beat a careful, systematic grid — but the picture above is the proof. Imagine only one of your two hyperparameters actually affects the score, and you have a budget of nine trials. A 3×3 grid tests that important hyperparameter at just three distinct values, because the other six trials merely repeat those three while fiddling the useless dimension. Random search tests it at nine distinct values. Same cost, three times the resolution on the thing that matters. Bergstra and Bengio's insight was that real problems almost always have this "low effective dimensionality" — a few hyperparameters dominate — so random search reliably wins. It became the default overnight, and it is still the sensible baseline before reaching for anything fancier.

Suppose you try 200 hyperparameter settings and, for each, check the accuracy on your test set, keeping the best. You have just used the test set 200 times to make a decision — so its score is no longer an unbiased estimate of anything. With enough settings, one will look great on that specific test set by pure luck, and you will ship it believing it is excellent. This is a subtle cousin of data leakage: the test data influenced the model, even though it was never in the training set. The cure is discipline — a separate validation set (or CV folds) for all tuning, and a test set that stays sealed until the single final evaluation. If you tune a lot, use nested cross-validation, where an inner loop tunes and an outer loop evaluates, so no data does double duty.

Learn this on Kaggle

Kaggle Learn's Intermediate Machine Learning course pairs cross-validation with hyperparameter search on real competition data, so you can watch grid and random search pick settings and measure the honest, cross-validated payoff. It is the natural place to practise everything on this page.