Missing Data and Imputation

Open almost any real dataset and you will find holes: a blank income field, a survey question skipped, a sensor that dropped out for an afternoon. Missing data is not a rare accident — it is the normal state of the world's data. And it is dangerous precisely because it is so easy to paper over. Delete the wrong rows, or fill the gaps with the wrong number, and you can quietly bias every conclusion that follows, without a single error message to warn you.

This lesson is about doing it thoughtfully. Two questions decide everything: why is the value missing, and what should you do about it?

The three mechanisms of missingness

Donald Rubin's classification, from 1976, is the vocabulary every data scientist shares. The crucial insight: whether it is safe to fill in a gap depends on why the gap is there.

Mechanism	Meaning	Example	Safe to ignore?
MCAR missing completely at random	Missingness is unrelated to anything — pure chance.	A test tube is dropped at random.	Yes — deletion is unbiased (just wasteful).
MAR missing at random	Missingness depends on observed variables, not the missing value itself.	Men skip a depression question more often — but you recorded sex.	Fixable — condition on the observed variables.
MNAR missing not at random	Missingness depends on the unseen value itself.	High earners decline to state their income because it is high.	No — dangerous; needs a model of the missingness.

The names are slippery, so hold onto the punchline: MCAR is a free lunch, MAR is recoverable if you use the columns you do have, and MNAR is the trap — the very fact that a value is missing is itself information, and no amount of clever filling fully rescues you.

Two responses: delete or impute

Listwise deletion (a.k.a. "complete-case analysis") simply drops any row with a missing value. It is honest and simple, and under MCAR it is unbiased — but it is greedy: if each of 20 columns is 5% missing and the gaps are scattered, you can lose over half your rows to the wastebin. Worse, under MAR or MNAR the survivors are a biased sample.

Imputation fills the holes instead. The simplest fill replaces each missing value with a summary of the column — the mean for a numeric feature, the median for a skewed one, or the mode for a category:

\hat{x}_{ij} = \bar{x}_j = \frac{1}{|O_j|}\sum_{i \in O_j} x_{ij},

where O_j is the set of rows that do have column j. It is one line of code and infinitely tempting — and it quietly damages your data in two ways at once.

What mean imputation does to a distribution

Every imputed value lands on the same spot — the mean — so the distribution grows a spike there. The real spread is squeezed and the relationships between variables are diluted. The picture below shows a feature's values before (top) and after (bottom) mean-imputing the gaps: the imputed points pile up in a single tower at the average.

Two mathematical facts explain the damage:

It shrinks the variance. Adding m copies of the mean to k real values leaves the mean unchanged but pulls the estimated variance down by roughly a factor of \tfrac{k}{k+m} — the more you impute, the more artificially "certain" the data looks.
It distorts correlations. Because the filled values ignore every other column, they drag relationships toward zero, weakening exactly the signals a model needs.

Better fills

Two ideas rescue most of the loss:

Model-based imputation predicts each missing value from the other columns — a regression, a k-nearest-neighbours average, or iterative methods like MICE that cycle through the features. This respects the correlations that mean imputation flattens.
The missing-indicator trick: alongside the filled column, add a binary flag m_{ij} \in \{0,1\} marking "was this value originally missing?" If missingness itself carries signal (hello, MNAR), the model can now learn from it instead of being fooled by it.

Surprisingly often, yes. In credit scoring, the fact that an applicant left "previous bankruptcies" blank can predict default better than any number they might have written. In medicine, a test that was never ordered tells you the clinician didn't think it necessary — which is itself a diagnosis. This is MNAR turned into an asset: the missing-indicator flag captures the signal that the blank was hiding. The lesson is to treat missingness as data, not just as a gap to be plastered over. Before you fill a column, always ask whether the pattern of holes is trying to tell you something.

The most seductive error is to mean-impute, then run your analysis as if the data had been complete all along. Your standard errors and confidence intervals will come out too narrow, because you have injected fabricated values that pretend to be real observations — the model has no idea those points were guessed. You end up more confident about a smaller amount of true information. The principled fix is multiple imputation: fill the gaps several times with slightly different plausible draws, analyse each completed dataset, and combine the results so that the extra uncertainty from not knowing the missing values is honestly propagated. A single fill throws that uncertainty away.

Learn this on Kaggle

Kaggle Learn's free Data Cleaning course has a whole lesson on handling missing values in real competition data, where you will drop, mean-fill and flag columns and watch the effect on a model's score — the ideas of this page turned into practice on messy files.