Data Leakage

A team builds a model to predict which hospital patients will develop sepsis. On held-out data it scores an astonishing 99% — nobody has ever seen numbers like it. They ship it. In production it is useless, barely better than a coin flip. What happened? One of the input features was "was the patient moved to the ICU" — something that only becomes true after a doctor already suspects sepsis. The model wasn't predicting the future; it was reading the answer off a feature that quietly encoded it.

This is data leakage: information about the target sneaking into the training data that would not be available at real prediction time. It is the single most common reason a model looks brilliant in development and collapses in the wild — and because the symptom is a suspiciously good score, it's the failure people are least inclined to question.

The three ways it sneaks in

Target leakage — a feature that is a proxy for, or a consequence of, the outcome, known only after the outcome is decided. "ICU transfer," "final invoice amount," "cancellation reason": each carries the answer. If a feature wouldn't exist yet at the moment you need the prediction, it must go.
Train/test contamination — fitting any data-dependent step (scaling, imputation, encoding, feature selection) on the whole dataset before splitting, so the test rows influence the transforms and secretly inform training.
Temporal leakage — in time-ordered data, training on rows from the future relative to what you're predicting. Shuffling a stock or sales dataset before splitting lets the model "remember" tomorrow while predicting today.

All three share one root cause: at training time the model had access to information it will not have at the moment of a real prediction. Leakage is best understood as breaking that one rule.

The contamination trap, concretely

Target leakage is usually caught by thinking hard about each feature. The sneakier one is train/test contamination, because it hides inside routine preprocessing. Suppose you standardize a feature by subtracting its mean and dividing by its standard deviation. If you compute that mean and standard deviation over the entire dataset and only then split into train and test, the mean has already been influenced by the test rows — the test set has leaked into the training features. The same happens when you:

impute missing values with a column mean computed over all rows;
target-encode a category using target values from the whole dataset;
select features by correlation with the target across every row;
oversample a minority class before splitting, copying test rows into the train set.

Fit every data-dependent transform on the training fold only, then apply the already-fitted transform to validation and test.
Inside cross-validation, that means re-fitting the transforms inside each fold — which is exactly what a pipeline object does for you automatically.

Leakage isn't a beginner's-only problem — it has decided real competitions. In one notorious Kaggle contest, the row ID turned out to be correlated with the target because the organisers had sorted the data by outcome before assigning IDs; a "feature" as innocent as the index leaked the answer. In another, a hospital's dataset let you predict cancer from the scanner ID, because the sickest patients had all been sent to one specialist machine. The lesson: leakage often lives in metadata nobody thinks of as a feature — IDs, timestamps, file paths, the order of the rows.

The instinct on seeing accuracy leap from 82% to 99% is to celebrate and ship. The trained instinct is to become suspicious. Enormous, sudden gains almost always mean leakage rather than genius — a feature that shouldn't be there, a transform fit on the wrong data, a shuffled time series. When a score looks too good to be true, audit the features one by one and ask of each: "would I actually know this value at the instant I need to make the prediction?" If the answer is no, the feature is leaking, and the impressive score is a mirage.

A checklist against leakage

Ask	If yes…
Is this feature known only after the outcome?	Target leakage — drop it
Did I fit scaling/imputation/encoding before splitting?	Contamination — move it inside the split/pipeline
Did I shuffle time-ordered data before splitting?	Temporal leakage — split by time instead
Is the score suspiciously high?	Audit every feature for a hidden proxy
Could an ID, timestamp or row order carry the answer?	Remove or investigate that metadata

Leakage is why the whole apparatus of honest evaluation — a locked test set, transforms fit inside folds, feature selection inside the loop — exists. Get the plumbing right and your development score finally means what you hope it means: how the model will do on data it has genuinely never seen.

Learn this on Kaggle

Kaggle Learn's Intermediate Machine Learning course closes with a lesson devoted entirely to data leakage, walking through both target leakage and train/test contamination on real examples — and showing how a scikit-learn pipeline stops preprocessing from leaking across the split. It is the single most valuable habit the course teaches.