Train, Validation, Test

You would never grade your own exam using the answer key you memorised it from — of course you'd score perfectly, and the score would tell you nothing about how well you actually understand the material. Machine learning has exactly the same trap, and it is one of the most common and most damaging mistakes in applied ML: judging a model using the same data it learned from, or — almost as bad — quietly reusing your "final test" so many times while tweaking the model that it stops being a fair test at all.

We already split data into train and test. But building a real model means making dozens of choices along the way — which features to use, how deep a tree should grow, how strong the regularization penalty \lambda should be. If you make those choices by checking the test set, you've quietly let it leak into the decision-making, and your final score becomes a lie you're telling yourself. The honest fix is to keep three separate, untouched piles of data:

Training set — what the model actually learns its parameters from.
Validation set — what you use to compare candidate models and settings while you're still deciding; you may check it as many times as you like.
Test set — locked away and touched exactly once, at the very end, for a final, honest verdict.

The workflow, in order

The three piles aren't just labels — they define a strict order of operations. First, train several candidate models (or the same model with several different settings) purely on the training set. Next, run every candidate on the validation set and compare their scores; this is where you pick the winner — the best value of k, the best \lambda, the best set of features. Only after the winner is chosen do you finally run it, once, on the test set. Whatever number comes out is your honest, reportable result — and it is also the last time that test set is ever allowed to influence anything.

Notice what this buys you: the validation set is allowed to shape your decisions precisely because you already know you'll double-check the final choice somewhere it has never been used to make decisions. Skip the test set, or use it more than once, and that safety net disappears.

The exam analogy actually maps onto the three sets almost perfectly. The training set is like the textbook and worked examples you study from — you're allowed to look at the answers as many times as you like while you're still learning. The validation set is like a practice exam: you take it, see your score, and use that feedback to decide what to restudy or which study method to switch to, over and over, right up until you feel ready. The test set is the real, final exam — sat exactly once, under conditions where you can no longer go back and "just check one more practice question" before submitting. Treating a practice exam as though it were the real one, over and over, tells you nothing honest about how you'd do on the day that actually counts.

Worked example: suppose you're deciding how strong to make a regularization penalty. You train five candidate models — \lambda = 0, 0.1, 0.5, 1, 5 — each purely on the training set, then measure all five on the validation set. Say \lambda = 0.5 comes out on top there. You now run only that one winning model on the test set, a single time, and report whatever number comes back — say 87% accuracy — as the model's true, honest performance. The other four candidates never touch the test set at all; they were eliminated using validation data alone.

Split the data three ways

Slide to change how much data goes to training; the rest is shared evenly between validation and test. A typical split for a modestly sized dataset is something like 60/20/20 or 70/15/15 — more training data usually means a better-fitted model, but you still need enough held back to tune confidently and to judge honestly at the end.

Put real numbers on it: a dataset of 10,000 labelled photos split 70/15/15 gives you 7,000 photos to train on, 1,500 to validate candidate models with, and 1,500 sealed away for the single final test. Shrink the dataset to just 1,000 photos with the same proportions, and the test set shrinks to 150 — still useful, but its accuracy estimate wobbles a lot more from one random split to the next, simply because there are fewer examples to average over.

The right ratio depends on how much data you have. With millions of examples, even a tiny slice — say 2% validation and 2% test — still leaves tens of thousands of examples for tuning and judging, so a lopsided 96/2/2 split is common and efficient, leaving the vast majority for training. With only a few hundred examples, the opposite problem appears: carve off 20% for test and you're judging your final model on a few dozen examples, which is noisy no matter how careful you are. That is exactly the situation cross-validation, described next, is built to help with.

Imagine you train a model, check its accuracy on your "test" set, don't like the number, go back and adjust a hyperparameter, and check the test set again. It feels harmless — you didn't touch the model's training data. But you did use information from the test set to steer a decision, and that is exactly what "leakage" means. Do this enough times and the test set has quietly become just another validation set — except you're still reporting its score as if it were an untouched, honest final number. It isn't anymore. Your reported accuracy will be optimistic, sometimes wildly so, and it will not hold up on genuinely new data. The rule has no exceptions: touch the test set exactly once, after every other decision has already been made.

Concretely: suppose a model genuinely performs at 75% accuracy on data it has never seen. If you tune twenty different settings by repeatedly checking the "test" set and keep whichever one scores highest on that same test set, you're no longer measuring the model — you're measuring the best of twenty rolls of the dice against that one particular batch of examples. It's entirely plausible to walk away reporting 85% or 90%, when the honest number, measured on truly fresh data, is still 75%. The gap doesn't show up until the model meets real data in production and quietly underperforms every claim made about it.

Size matters too, in the other direction. A test set that is too small gives a final verdict that bounces around a lot depending on which exact examples happened to land in it — a handful of unlucky or lucky cases can swing the reported accuracy by several percentage points. A trustworthy final number needs both an untouched test set and enough examples in it to be stable.

Cross-validation, when data is scarce

With little data, carving off a single validation set feels wasteful — every example set aside for validation is one fewer example the model gets to learn from — and the resulting score can be noisy, since it depends on which particular examples happened to land in that one slice. k-fold cross-validation fixes both problems at once: split the training data into k equal folds, then train k separate times, each time validating on a different fold and training on the rest. Averaging the k resulting scores gives a far steadier estimate of how a setting really performs — every example gets used for both training and validation at some point, just never both at the same time. The test set still stays sealed away, untouched, until the very end, exactly as before; cross-validation only changes how the training/validation portion is managed.

Training set — used to fit a model's parameters (its weights).
Validation set — used to compare candidate models or hyperparameter settings and choose among them; may be consulted repeatedly during development.
Test set — used exactly once, after all choices are finalized, to report an unbiased estimate of performance on unseen data.
k-fold cross-validation replaces a single fixed validation set with k rotating folds drawn from the training data, averaging k scores; the test set's role is unchanged.

Worked example: say you've already carved off a test set, leaving 1,000 examples for training-plus-validation. With k = 5 folds, those 1,000 examples split into five groups of 200. Round one trains on folds 2–5 (800 examples) and validates on fold 1 (200 examples); round two trains on folds 1, 3, 4, 5 and validates on fold 2; and so on, five rounds in total. Average the five validation scores together and you get a single, much steadier estimate of how well a given setting performs — built from all 1,000 examples, without ever letting any of them serve as both trainer and judge in the same round.

Real machine-learning competitions enforce the train/validation/test split more ruthlessly than any classroom ever could. On a platform like Kaggle, competitors download a training set and can check their own validation scores as often as they like — but the true test set's labels are withheld entirely, often on a server nobody can see, right up until the closing bell. You submit predictions, not code that could peek, and your leaderboard score is computed on data you genuinely never touched. That is the whole design: it makes test-set leakage structurally impossible, not just against the rules.

Outside of competitions, the discipline sometimes slips. A number of published research results have later turned out to be less impressive than they first looked, because — often by an honest accident during data preparation — a handful of test examples ended up duplicated into the training set, or a preprocessing step was fit using statistics from the whole dataset, test set included, before the split was made. The lesson is the same one this page keeps returning to: the moment test data influences any decision, however indirectly, the final number it produces can no longer be trusted.

One especially sneaky version of this mistake is easy to miss: fitting a preprocessing step — like scaling features to a common range, or filling in missing values with the dataset's average — using all the data, including the test rows, before splitting. Even though the model itself never sees a test label during training, the scaling numbers it depends on were quietly computed with the test set's help. The fix is simple once you know to look for it: always fit preprocessing steps on the training set alone, then apply those same fitted numbers — unchanged — to the validation and test sets.