Feature Scaling and Transformations

Suppose you predict house prices from two features: number of bedrooms (say 1–6) and floor area in square metres (say 30–400). To a distance-based model these live on wildly different scales — a one-bedroom difference is a rounding error next to a 200 m² gap. Left raw, the big-numbered feature drowns out the small one purely because of its units, not its importance. Rescale square metres to square feet and the model's behaviour changes, which should feel alarming: the physics of a house didn't change, only the ruler.

Feature scaling puts every feature on a comparable footing, and feature transformations reshape a feature's distribution so a model can use it well. Both are cheap, both are often decisive, and both hide a subtle leakage trap we will name at the end.

Two ways to rescale

The two workhorses both map a feature onto a standard range, differently:

Standardization (z-score): subtract the mean, divide by the standard deviation, giving a feature with mean 0 and standard deviation 1. z = \frac{x - \mu}{\sigma} The transformed values are z-scores — "how many standard deviations above or below the mean." It doesn't bound the range, so outliers survive as large z-values.
Min-max normalization: linearly squeeze the feature into [0,1]. x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}} Neat and bounded, but a single extreme outlier sets x_{\max} and crushes everyone else toward 0.

A rough rule: reach for standardization when the feature is roughly bell-shaped or you're feeding a linear/kernel model, and min-max when you need a hard bounded range (e.g. image pixels, or a neural net input layer). For heavy-tailed data with outliers, a robust scaler (subtract the median, divide by the interquartile range) beats both.

Which models care — and which don't

Scaling matters exactly when the model measures distances or takes gradient steps in the raw feature space:

Model	Needs scaling?	Why
k-nearest neighbours, k-means	Yes	Euclidean distance is dominated by the largest-scale feature
SVM (RBF/linear), regularised regression	Yes	The kernel and the penalty treat all coordinates alike
Gradient descent (neural nets, logistic)	Yes	Unequal scales make loss contours elongated → slow, zig-zag convergence
Decision trees, random forests, gradient boosting	No	Splits use per-feature thresholds; monotone rescaling changes nothing

This is why you'll see pipelines standardize before a gradient-descent regression but skip it entirely before a random forest: a tree asks "is area > 120?", and the answer is identical whether area is in metres, feet, or z-scores.

Transformations: reshaping a skewed feature

Scaling shifts and stretches a feature but keeps its shape. Many real features — income, population, prices, counts — are heavily right-skewed: a long tail of huge values drags the mean around and swamps a linear model. A log transform (x \mapsto \log(x+1)) compresses that tail and pulls the distribution toward symmetry, so relationships become closer to linear and outliers stop dominating. The chart shows the idea — a right-skewed feature, and its far more symmetric shape after a log:

Box-Cox and Yeo-Johnson generalise this: they search a family of power transforms for the exponent \lambda that makes the feature as Gaussian as possible (Box-Cox needs positive values; Yeo-Johnson also handles zero and negatives). Two more reshapers round out the toolkit:

Binning / discretization — chop a continuous feature into buckets ("age 0–17, 18–34, …"). It throws away resolution but can capture non-linear effects and tame outliers, and it turns a number into a category you might then encode.
Polynomial & interaction features — add x^2, x^3, or a product x_1 x_2 so a linear model can bend and can see that two features together matter (area × neighbourhood). Powerful, but the column count grows fast — a reason to scale afterwards and to select carefully.

Picture the loss as a bowl over the weight space. If one feature ranges over thousands and another over units, the bowl is a long, thin valley: the gradient points mostly across the valley, not along it, so the optimiser bounces from wall to wall and inches toward the minimum. Standardizing makes the bowl round, the gradient points straight downhill, and the same learning rate converges in a fraction of the steps. It's the single cheapest speed-up in classical ML.

The mean, standard deviation, min, max, or Box-Cox \lambda a transform needs are parameters learned from data — and if you learn them from the whole dataset before splitting, the test set has silently influenced the training features. That is data leakage, and it inflates your reported score. The rule is unbending: fit the scaler on the training fold, then apply those same fitted numbers to the validation and test sets. In practice you wrap the scaler in a pipeline so cross-validation re-fits it on each fold automatically.

Learn this on Kaggle

Kaggle Learn's Feature Engineering course has you build log transforms, interaction terms and scaled pipelines on real data, and shows how scikit-learn's pipeline objects keep scaling from leaking across the split. Try transforming a skewed price column and watching a linear model improve.