Class Imbalance

Roughly 1 card transaction in 10,000 is fraudulent. About 1 visitor in 200 buys anything. Some cancers appear in a handful of patients per thousand screened. These are imbalanced problems: the thing you actually care about — the fraud, the sale, the disease — is a rare positive drowning in a sea of ordinary negatives. And imbalance quietly breaks the most natural way of scoring a model.

Suppose 1 in 1,000 transactions is fraud. A "model" that shrugs and labels everything legitimate is right 999 times out of 1,000 — a dazzling 99.9% accuracy — while catching precisely zero frauds. Accuracy has told you the model is superb when it is worthless. On a rare-positive problem, accuracy is not just unhelpful; it is actively misleading, because the trivial "always say no" baseline already scores near the top.

Use the metrics that watch the rare class

The escape is to stop scoring the whole dataset and instead score how well you handle the positives, using the tools from accuracy, precision and recall:

Recall = \tfrac{TP}{TP+FN} — of all the real positives, how many did we catch? The trivial "always negative" model scores 0 here, exactly as it deserves.
Precision = \tfrac{TP}{TP+FP} — of everything we flagged, how much really was positive? This is where rarity bites hardest, as we'll see.
F1 = \tfrac{2\,\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}} — the harmonic mean, a single number that stays low unless both are decent.
PR-AUC — the area under the precision–recall curve (also called average precision). Unlike ROC-AUC, it feels every false alarm, so it is the honest summary when positives are rare.

ROC-AUC deserves a special warning here. Its false positive rate divides by the huge number of negatives, so a flood of false alarms barely nudges it — a fraud model can post ROC-AUC 0.99 and still bury analysts in false positives. The precision–recall curve divides by the number of predicted positives instead, so it cannot be flattered this way. On imbalanced problems, prefer PR-AUC.

Why precision collapses when positives are rare

Recall depends only on the positives, so rarity doesn't touch it. Precision is different — and the algebra shows exactly why. Let \pi be the prevalence (the fraction of cases that are truly positive), r the recall, and f the false positive rate. Then out of any batch of cases, the true positives number \pi r and the false positives number (1-\pi) f, so:

\text{precision} = \frac{\pi\, r}{\pi\, r + (1-\pi)\, f}.

Now let the positives get rare, \pi \to 0. The (1-\pi)f term in the denominator is anchored near f while the \pi r terms shrink toward zero, so precision collapses no matter how good the recall is — even a perfect recall of r = 1 cannot save it. The false alarms come from the enormous negative class, and there are simply far more negatives to mislabel than positives to find. Slide the prevalence and watch precision fall off a cliff while recall and the false-positive rate never move:

This is why "my model has 95% recall" can hide a precision of 8% on a rare-positive problem: it catches nearly every fraud, but nine of every ten alerts it raises are false. Reporting recall alone on imbalanced data is as misleading as reporting accuracy.

Four ways to fight back

Once you are measuring the right thing, several remedies push a model to take the rare class seriously. They fall into three families plus a free knob.

Fix	What it does	Watch out for
Oversampling (incl. SMOTE)	Duplicate or synthesise extra positive examples so the classes look balanced during training. SMOTE builds new positives by interpolating between existing ones rather than copying.	Naïve copying can overfit the exact positives; SMOTE can invent implausible points.
Undersampling	Throw away most negatives to balance the classes — fast and simple.	Discards real data, losing information about the negatives.
Class weights	Leave the data alone; instead weight the loss so each missed positive costs far more than a missed negative.	You still have to choose the weight ratio sensibly.
Threshold moving	Keep the model, but lower the decision threshold below 0.5 so borderline cases count as positive.	Trades precision for recall — pick the point that matches real costs.

Class weights and threshold moving are usually the first things to try, because they change nothing about the data itself — they just tell the model, or the final cutoff, how much you dread a missed positive. Resampling is heavier machinery, and SMOTE in particular is a popular but double-edged tool.

The simplest oversampling just duplicates positive rows until the classes match. But a model shown the same fraud case fifty times can memorise that exact point rather than learning what fraud looks like in general — you have inflated the count without adding any new information. SMOTE (Synthetic Minority Over-sampling Technique) is smarter: for each rare example it finds a near neighbour of the same class and creates a brand-new synthetic point somewhere on the line between them. The result is a plausible region of positives rather than a few points repeated, which helps the model draw a smoother boundary around the rare class. The catch, as ever, is that interpolating blindly can manufacture "positives" in places no real positive would ever fall — so SMOTE is powerful but not automatic magic.

The single most common imbalance mistake is oversampling or undersampling the whole dataset before splitting off a test set. Do that and your test set no longer reflects reality: it has been rigged to look balanced, your metrics look wonderful, and the model faceplants in production where fraud is still 1 in 10,000. Balancing is a training-time trick only. Always split first, then resample inside the training fold alone, and always evaluate on data that keeps the true, ugly, imbalanced prevalence. The same rule bites with cross-validation: the resampling must happen inside each fold, never before the folds are cut, or positives will leak between train and validation.

Learn this on Kaggle

Kaggle Learn's Intermediate Machine Learning course puts you on real datasets where the positive class is scarce. Practise scoring with precision, recall and F1 instead of accuracy, and try class weights and threshold moving to see the trade-offs move.