Feature Selection and Mutual Information

More features feel like more information, and more information feels like a better model. It isn't so. Past a point, extra features make models worse: they add noise, invite overfitting, cost compute and money to collect, and turn an interpretable model into an inscrutable one. The discipline of choosing a small, potent subset is feature selection, and this page builds up to its sharpest single tool — mutual information, a measure of dependence that sees relationships a correlation coefficient is blind to.

Why fewer, better features win

Four forces push toward a lean feature set:

The curse of dimensionality. As dimensions grow, data points spread thin and "nearest" neighbours stop being near — distance-based models degrade and you need exponentially more data to fill the space. We met this in the curse of dimensionality.
Overfitting. Every irrelevant feature is another chance for the model to latch onto noise that happens to look like signal in the training data.
Interpretability. A model with five meaningful features can be explained to a doctor or a regulator; one with five thousand cannot.
Cost. Each feature must be collected, cleaned, stored and served at prediction time — a feature that adds 0.1% accuracy but requires a slow database call may not be worth it.

Three families of method

Selection methods differ in how tightly they couple to the model you'll eventually train:

Family	How it works	Trade-off
Filter	Score each feature against the target with a cheap statistic (correlation, χ², mutual information) and keep the top ones — model-agnostic	Fast and scalable, but judges features one at a time, so it misses features useful only in combination
Wrapper	Search over feature subsets, actually training the model on each (forward selection, backward elimination, recursive feature elimination)	Catches interactions and tailors to the model, but expensive — many model fits
Embedded	Selection happens inside training: L1/Lasso drives weak coefficients to exactly zero; trees expose feature importances	Efficient and model-aware, but tied to that particular model family

The embedded star is L1 (Lasso) regularisation: adding a penalty proportional to the sum of absolute weights \lambda \sum_j |w_j| pushes the least useful coefficients all the way to 0, so the model selects features as a by-product of fitting. (The L2/ridge penalty shrinks weights but never zeroes them — it doesn't select.)

Mutual information: dependence beyond correlation

The best filter score for general use is mutual information. Built from entropy, it asks: how much does knowing feature X reduce our uncertainty about the target Y?

I(X; Y) = H(Y) - H(Y \mid X).
It is the entropy of Y minus the entropy of Y that remains once you know X — the uncertainty X removes.
It is symmetric (I(X;Y) = I(Y;X)) and non-negative, and it is exactly the KL divergence between the joint distribution and the product of the marginals — 0 precisely when X and Y are independent.

Its superpower is that it captures any statistical dependence, not just a straight-line one. Pearson correlation measures only linear association: a feature and target related by Y = X^2 over a symmetric range have correlation near zero — the classic case where correlation declares "no relationship" while the two are, in fact, perfectly determined. Mutual information sees straight through that: knowing X pins down Y completely, so I(X;Y) is large. For detecting non-linear, non-monotone relationships in a filter step, it is the tool to reach for.

Two catches. First, estimating mutual information from a finite sample — especially for continuous features — is fiddly: it needs binning or a nearest-neighbour estimator, and the estimate is noisy with little data. Second, like every filter it scores features one at a time against the target, so it can keep two highly redundant features that each look great alone but add nothing together, and it can drop a feature that is useless alone yet decisive in combination (the classic XOR pattern). "Maximum relevance, minimum redundancy" (mRMR) methods extend mutual information to fight the first problem; wrapper and embedded methods handle the second.

The most common way to fool yourself with feature selection is to run it on the whole dataset first — picking the features that correlate best with the target across every row — and only then split into folds to evaluate. But the selection has already peeked at the test rows' targets, so the reported score is optimistic, sometimes wildly so. This is data leakage in disguise. Feature selection is part of model training and must happen inside each cross-validation fold, using only that fold's training data — never on the held-out rows.

Learn this on Kaggle

Kaggle Learn's Feature Engineering course has a whole lesson on mutual information, where you compute MI scores with scikit-learn to rank features on a real housing dataset before you engineer new ones. It's the fastest way to make this measure concrete.