Recommender Systems

Roughly a third of what people watch on Netflix and buy on Amazon comes not from search but from a recommendation — "because you watched…", "customers also bought…". A recommender system is a data product whose whole job is to guess, from a mountain of past behaviour, the handful of items you in particular will love next. It is one of the highest-value applications in all of data science, and it rests on a single elegant idea: taste is low-dimensional. Millions of users and items can be explained by a few hidden "factors."

Two philosophies

Content-based systems are transparent but boxed in by their features; collaborative systems discover surprising, cross-genre gems but need lots of overlapping history. Most real systems are hybrids that fuse both.

The rating matrix, and why it's mostly holes

Lay every user's ratings in a giant table: rows are users, columns are items, and the entry R_{ui} is how user u rated item i. The defining feature of this matrix is that it is overwhelmingly empty — any one person has rated a vanishing fraction of a catalogue of millions. Ninety-nine-point-something percent of the cells are blank. The recommender's task, stated crisply, is to fill in the blanks: predict the ratings that aren't there, then recommend the items with the highest predicted score.

Matrix factorization: the low-rank secret

The breakthrough that won the famous Netflix Prize was matrix factorization. Assume the sprawling ratings matrix R is secretly the product of two thin matrices:

R \;\approx\; U\,V^{\top}

Here each user u gets a short vector of k latent factors (a row of U), each item gets its own k-vector (a row of V), and the predicted rating is simply their dot product, \hat{R}_{ui} = \mathbf{u}_u \cdot \mathbf{v}_i. The factors are learned, not labelled, but they often turn out to mean something — one axis might track "gritty ↔ feel-good," another "blockbuster ↔ arthouse." A user who scores high on "arthouse" is matched to items that also score high on it. This is the very same singular value decomposition idea — approximate a big matrix by a low-rank product — put to work on taste.

We find U and V by minimising the squared error on the ratings we do have (plus a regulariser to avoid overfitting the sparse data):

\min_{U,V}\;\sum_{(u,i)\,\in\,\text{observed}} \big(R_{ui} - \mathbf{u}_u \cdot \mathbf{v}_i\big)^2 \;+\; \lambda\big(\lVert U \rVert^2 + \lVert V \rVert^2\big)

Crucially the sum runs only over observed cells — we never ask the model to explain the blanks, only to fit the known ratings well enough that its predictions for the blanks are trustworthy.

Cold start and implicit feedback

Two realities complicate the tidy picture. Cold start: a brand-new user or item has no ratings, so collaborative filtering has nothing to go on — the classic fix is to fall back on content features (or ask a few onboarding questions) until history accumulates. Implicit feedback: most systems never get star ratings at all; they get clicks, watches, purchases, dwell time. That signal is one-sided — a watch is a weak "like," but a non-watch is not a "dislike," it might just be "never saw it." Modelling implicit feedback well (weighting confidence, treating absence carefully) is where much of the real engineering lives.

In 2006 Netflix offered $1,000,000 to anyone who could beat its in-house recommender by 10% on prediction error. The three-year contest became a landmark in machine learning: matrix-factorization methods, previously niche, proved dominant and entered the mainstream, and the winning entry was a giant ensemble blending dozens of models. There's a twist worth remembering — Netflix never deployed the winning solution. It was too complex to run in production for the marginal gain, and the business had already pivoted from DVDs toward streaming, where implicit signals mattered more than the star ratings the prize optimised. A perfect lesson that the offline metric is not the business.

A recommender does not passively observe taste — it shapes what users see, which shapes what they click, which becomes tomorrow's training data. This feedback loop can trap the system: popular items get recommended, so they get clicked, so they look even more popular, while good obscure items never get a chance to be discovered — a rich-get-richer "popularity bias." Worse, users can be walled into a filter bubble of ever-narrowing sameness. The metric on your held-out log looks great precisely because the log was generated by the old recommender's choices. Real systems must deliberately inject exploration (occasionally recommending uncertain items) and be evaluated online, not just on their own self-fulfilling history.

Learn this on Kaggle

Kaggle doesn't have a course dedicated solely to recommenders, but Kaggle Learn's Intermediate Machine Learning course covers the modelling and validation foundations, and the community hosts many excellent recommender-system notebooks — search Kaggle for "collaborative filtering" or "matrix factorization" to build one on a real ratings dataset in Python.