Cross-Validation

You hold out a test set, train, and score 91%. But was that 91% real, or did you get lucky with which rows happened to land in the test set? With a single split you can't tell — and if the test set is small, the score wobbles alarmingly from one random split to the next. A model that looks great on one split can look mediocre on another, and you'd never know.

Cross-validation fixes this by not trusting any single split. It rotates the held-out portion through the whole dataset, so every row gets a turn being evaluated, and averages the scores into an estimate that is far more stable. It is the backbone of honest model evaluation, built on top of the train / validation / test split.

k-fold cross-validation

Split the data into k equal parts, or folds. Then run k rounds: each round holds out one fold to score on and trains on the other k-1. Every fold is the validation set exactly once, and the final estimate is the average of the k scores. The diagram shows k = 5: each row is one round, the highlighted block is that round's held-out fold.

With k = 5 or k = 10 (the usual choices) every row contributes to the score, so you use the data far more efficiently than a single hold-out — and the average of five or ten estimates has much less variance than any one of them. The price is compute: you train the model k times instead of once.

Three variations you'll actually use

Stratified k-fold — for classification, especially imbalanced classes, build each fold to preserve the overall class ratios. Without stratification a rare class can be absent from a fold entirely, making that round's score meaningless.
Leave-one-out (LOOCV) — the extreme k = n: each fold is a single data point. It uses almost all the data to train every time (low bias) but the n models are nearly identical and their errors are highly correlated (high variance in the estimate), and it costs n fits — brutal on large data.
Nested cross-validation — when you also tune hyperparameters, wrap an inner CV loop (which picks the hyperparameters) inside an outer CV loop (which scores the whole tuning-plus-training procedure). The outer score stays honest because the data used to choose settings never scores them.

The choice of k is itself a bias–variance tradeoff, applied to the estimate rather than the model: small k (say 2 or 3) trains on less data, biasing the estimate pessimistically; large k trains on nearly all the data (low bias) but the highly overlapping training sets make the fold scores correlated, raising the variance of their average. k = 5 or 10 is the sweet spot precisely because it balances the two.

It feels backwards — surely training on n-1 points every time should give the most reliable estimate? The subtlety is that LOOCV's n training sets differ by only a single point, so the n fitted models are almost the same model, and their errors are strongly correlated. The average of many correlated numbers doesn't shrink its variance the way the average of independent numbers does. Ten-fold CV, whose training sets overlap less, produces less correlated fold scores and so a lower-variance average — which is a large part of why practitioners rarely bother with LOOCV.

Cross-validation is for choosing and tuning your model — comparing algorithms, picking hyperparameters. The moment you use CV scores to make decisions, those folds have informed your choices, so a good CV score is not an unbiased estimate of future performance. You still need a final, untouched test set, locked away until the very end, to get an honest number. Reporting your best cross-validation score as the model's real-world accuracy is a classic way to overstate it — the test set exists precisely to keep you honest after all the tuning is done.

Learn this on Kaggle

Kaggle Learn's Intermediate Machine Learning course has a dedicated cross-validation lesson where you run k-fold with scikit-learn's cross_val_score and see the fold-to-fold spread for yourself. It's the quickest way to feel why one split isn't enough.