Suppose you're predicting a house's price from two features: its number of bedrooms (typically 1 to 5) and its size in square feet (typically 500 to 5000). Both are perfectly reasonable numbers — but they live on wildly different scales. A change of "1" means almost nothing for square footage and everything for bedroom count.

Gradient descent updates every weight using the very same learning rate. If bedroom counts sit near 1–5 and square footage sits near 500–5000, a single learning rate that's sensible for the small-numbered feature is far too tiny for the big-numbered one (learning would crawl), while a rate sized for the big feature would send the small feature's weight wildly overshooting. Scaling first means one learning rate can serve every feature well.

Feed both features straight into gradient descent and that mismatch warps the cost surface into a long, narrow, elongated valley rather than a nice round bowl. Descent has to creep along the narrow direction while over-correcting in the wide one, so it zig-zags painfully slowly toward the bottom instead of heading straight there.

The fix: put every feature on the same footing

Feature scaling rewrites every feature so they all sit in a similar range before training even starts. Two rescalings cover almost every case you'll meet:

Either way, every feature ends up living on a comparable scale, so no single feature's raw units can dominate just because its numbers happen to be bigger.

Worked example: min-max scaling bedroom counts

A tiny dataset of bedroom counts: 1, 2, 3, 4, 5. Here x_{\min}=1 and x_{\max}=5, so x_{\max}-x_{\min}=4. Apply x' = (x-1)/4 to each value:

Notice the smallest raw value always lands on 0 and the largest always lands on 1 — that's the whole point of min-max scaling.

Worked example: standardizing square footage

Now take three raw house sizes — 900, 1800, and 2700 square feet — and suppose the training set's mean size is \mu = 1800 with standard deviation \sigma = 650. Standardize with x' = (x-\mu)/\sigma:

The house exactly at the mean scales to 0; a house one "spread" below the mean scales to about -1.385. After standardizing, bedroom counts and square footage — however different their raw units — both cluster in roughly the same small range around zero.

Putting both features together

Here's the original hook, worked all the way through. Three houses, each described by (bedrooms, size in sq ft): (2, 900), (3, 1500), and (5, 2100). Standardize each column separately — bedrooms have mean \mu=3.33, \sigma\approx1.25; size has mean \mu=1500, \sigma\approx490:

Before scaling, "size" ranged over hundreds while "bedrooms" ranged over single digits — size would have swamped bedrooms in any distance or gradient calculation. After scaling, both columns sit comfortably in roughly [-1.5, 1.5], exactly the "similar range" feature scaling promises.

Which one should you reach for?

Min-max scaling and standardization both fix the same problem, but they behave differently at the edges of the data:

• Min-max scaling guarantees every value lands neatly in [0,1], which is handy when a downstream method expects bounded inputs. Its weak spot is outliers: one unusually huge house (say, a 20,000 sq ft mansion in an otherwise normal dataset) becomes the new x_{\max} and squashes every ordinary house's scaled value down near 0, throwing away most of the resolution you actually cared about.
• Standardization has no guaranteed bounds — a very large value can still produce a large z-score — but because it scales by the typical spread rather than the single most extreme value, one outlier does far less damage to everyone else's scaled numbers. It's the more common default for gradient-descent-trained models for exactly this reason.

Either way, the recipe for a brand-new example is the same: never recompute x_{\min}, x_{\max}, \mu, or \sigma from it. Take the numbers already saved from training, plug the new raw value straight into the same formula, and use whatever falls out — even if it lands slightly outside [0,1] for min-max, or a bit further than usual from 0 for standardization. That's expected and correct: it just means the new house is a bit smaller or bigger than anything seen during training.

Stretched valley vs round bowl

Back to the cost surface. The rings below are contours of the cost — points of equal cost, like a topographic map. On unscaled features (bedrooms vs. square feet, raw) the contours are stretched ellipses, and gradient descent's path skitters side to side as it creeps down the narrow axis. On scaled features the contours are near-circles and the same algorithm drives almost straight to the centre. Same data, same algorithm, same learning rate — the only thing that changed is the scale of the inputs.

This is exactly the bowl-shape story from visualizing the cost: a round bowl lets every step point roughly at the minimum, while a squashed one wastes most of each step correcting sideways instead of making progress downhill.

A cheap habit that always pays

Scaling costs almost nothing to compute and can turn a training run from painfully slow (or one that never quite converges) into a quick, well-behaved descent. Because it's cheap and the upside is large, most practitioners simply do it by default before running gradient descent on any dataset with more than one feature.

It's worth knowing scaling is specifically a gradient descent concern: the normal equation solves for the best weights directly, in one shot, without taking iterative steps at all — so it isn't fighting a stretched valley the way descent is. Scaling can still help it numerically on tricky data, but the dramatic "zig-zag vs straight line" story above is entirely about how gradient descent walks downhill.