The Feature Vector

Every single example a machine-learning model ever sees — a house, an email, a photo, a song — gets boiled down to the same thing: an ordered list of numbers. No matter how messy or real-world the original object is, by the time it reaches the model it has become a vector — the feature vector. That translation, from "real thing" to "list of numbers in a fixed order," is what makes the whole of linear algebra — dot products, distances, matrices — usable on data at all.

Bundle an example's features together, in a fixed order, and that's the feature vector. A house described by (\text{area}, \text{bedrooms}, \text{age}) becomes

\vec{x} = \begin{bmatrix} 90 \\ 3 \\ 12 \end{bmatrix}.

This is the quiet bridge between machine learning and linear algebra: every example in a dataset is a point in feature space, and the whole dataset is a cloud of such points.

Worked example: turning a house into a vector

Suppose the model needs to predict a house's price. We pick three features — floor area in square metres, number of bedrooms, and age in years — and read them off one real house:

Line those three numbers up, in that order, in a column, and the house is the vector \vec{x} = (90, 3, 12). Nothing clever has happened yet — we've simply agreed on an order and written the numbers down — but that's the entire trick. The house's price (what we're trying to predict) is the label, kept separate from the vector; the vector is only ever the inputs.

Worked example: turning an email into a vector

Text is messier than a house's measurements, but the same trick works. Pick a short list of "key" words to watch for — say free, money, meeting, lunch — and count how many times each one appears in an email. This is called a bag of words: word order and grammar are thrown away, and all that survives is how often each chosen word shows up.

An email that reads "FREE FREE money now, don't miss this free offer!" would count as free: 3, money: 1, meeting: 0, lunch: 0 — giving the feature vector

\vec{x}_{\text{spam}} = \begin{bmatrix} 3 \\ 1 \\ 0 \\ 0 \end{bmatrix}.

A calmer email — "Let's grab lunch before the meeting" — counts as free: 0, money: 0, meeting: 1, lunch: 1, giving

\vec{x}_{\text{normal}} = \begin{bmatrix} 0 \\ 0 \\ 1 \\ 1 \end{bmatrix}.

Two wildly different emails, each reduced to four numbers a spam filter can compare, measure the distance between, or feed straight into a dot product.

The single most dangerous bug in this whole idea is a silent one: swapping which slot means what. Say a house model is trained expecting (\text{area}, \text{bedrooms}, \text{age}) — position 2 always means "bedrooms." If, later, the code that builds vectors for new houses accidentally puts age in slot 2 and bedrooms in slot 3, nothing crashes. There's no error message. The model happily multiplies its "bedrooms" weight against what is actually the house's age, and quietly returns a garbage prediction that looks perfectly plausible.

This is why feature order is treated as a strict contract: feature #2 must mean exactly the same thing for every example in the dataset, and for every new example fed to the model afterwards — training time and prediction time alike. Real ML pipelines pin this down with a fixed schema (a documented, ordered list of feature names) precisely so that "slot 2" can never quietly drift in meaning.

A feature vector is a list of numbers — but plenty of real features are categories, not numbers: a house's neighbourhood, a car's colour, a customer's country. You can't just write "blue" into a numeric vector. Categorical features have to be converted into numbers first.

The simplest fix — numbering the categories 0, 1, 2, \dots — is usually a trap, because it invents a false ordering and false distances (is "green" really "between" red and blue?). The standard trick instead is one-hot encoding: give each category its own slot that is 1 when the example is that category and 0 otherwise. "Red" among {red, blue, green} becomes (1, 0, 0); "blue" becomes (0, 1, 0). Three extra numbers, no false ordering — and now it's a perfectly ordinary chunk of the feature vector.

An example is a point

With two features we can actually see it: each example is a dot in the plane, and equally an arrow from the origin. Slide the two features and watch the point move through feature space. Real datasets have hundreds of features — hundreds of dimensions — but the idea is identical; we simply lose the ability to draw it.

A third feature — into 3D

Add a third feature and the example becomes a point in three-dimensional feature space. Drag the box to rotate it, and slide the three features to move the point. A dashed line drops the point onto the feature-1 / feature-2 floor: the shaded rectangle there reads off those two coordinates, and the drop's height gives the third. Past three features we can no longer draw it, yet nothing in the maths changes: distances and dot products work the same in 3D, 100D, or 10,000D.

Why vectors, not just lists

Calling it a vector — not merely a list — is the point. It lets a model measure things: the distance between two examples (are they similar?), and the dot product of an example with a set of weights (its score). Those two operations — distance and dot product — are the seeds of nearly every algorithm ahead.

A recommendation engine deciding what to show you next, a spam filter reading your inbox, and an image classifier spotting a cat in a photo have almost nothing in common on the surface — one reads clicks, one reads words, one reads pixels. Yet under the hood, every one of them does exactly the same first step: turn the input into a feature vector. Songs become vectors of listening statistics, emails become vectors of word counts, images become vectors of pixel brightnesses. The feature vector is the universal translation layer between the messy real world and the tidy world of linear algebra — every model, however different its job, starts by speaking this one common language.

The one genuinely hard part nobody automates for you is which features to bother collecting in the first place — as the "good vs. useless feature" question over in features and labels explored. Picking good features is often more art than science; building the vector out of them, once chosen, is the easy, mechanical part.