In 2018 a large tech company quietly scrapped an experimental résumé-screening model. It had been trained on a decade of the company's own hiring decisions — and since those hires had skewed heavily male, the model dutifully learned that being male was a positive signal. It penalised résumés containing the word "women's" (as in "women's chess club captain") and downgraded graduates of two women's colleges. Nobody wrote a rule to discriminate. The data carried the bias, and the model faithfully amplified it. This is the central problem of algorithmic fairness: a model is a mirror of its training data, and if that mirror reflects an unjust past, the model will project it into the future — at scale, with a veneer of objectivity.
Bias is rarely a single villain; it enters through several doors, often at once.
The hard surprise is that fairness is not one definition — it is several, and they conflict. Three of the most-used, for a model deciding who gets a loan across two groups:
These sound compatible until you look closely. The now-famous impossibility result proves that when the underlying base rates differ between groups, you cannot satisfy all of these at once — demographic parity and equalized odds are mathematically at odds. Choosing which fairness criterion to enforce is therefore an ethical and political decision, not a technical one the data can settle for you.
The law often cares less about intent than about outcome. A policy shows disparate impact when it is neutral on its face yet falls much harder on a protected group in practice. In US employment law a common yardstick is the four-fifths rule: if one group's selection rate is below 80% of the most-favoured group's, that is prima-facie evidence of adverse impact. A model can be blind to race by construction and still produce a starkly disparate impact through proxies — which is exactly why "we didn't use the protected attribute" is not a defence, legally or ethically.
In 2016 a widely-used tool that scored a criminal defendant's risk of reoffending was accused of racial bias: among defendants who did not go on to reoffend, Black defendants were labelled high-risk far more often than white ones — a false-positive-rate gap, violating equalized odds. The tool's maker replied that the scores were calibrated — a given risk score meant the same actual reoffending probability regardless of race, another reasonable notion of fairness. Astonishingly, both sides were right. Researchers proved that when base rates differ, calibration and equal false-positive rates cannot both hold. The case is the canonical demonstration that "is this algorithm fair?" has no single answer — you must first say which fairness you mean, and accept that guaranteeing one can forbid another.
The most common and most dangerous fairness mistake is "fairness through unawareness": drop the race or sex column and declare the model neutral. It is almost never neutral. In a rich dataset, the protected attribute is redundantly encoded across dozens of correlated features — postcode, name, purchase history, even writing style — so the model happily reconstructs it and discriminates just as before, now invisibly. Worse, you have thrown away the very column you would need to measure and correct the disparity. Real fairness work requires the opposite: keep the sensitive attribute for auditing, measure outcomes across groups, and intervene deliberately — not pretend the variable doesn't exist.
Fairness cannot be bolted on at the end by a compliance team; it is decided by choices made all along the pipeline — what data to collect, which features to include, what metric to optimise, which fairness criterion to demand, and how to monitor the deployed system for drift. The data scientist is the one person positioned to see all of these at once. Treating a model's disparate outcomes as "just what the data said" is an abdication: the data said it because of decisions humans made, and different decisions yield a different model.
Kaggle Learn's Intro to AI Ethics course covers bias, fairness criteria and disparate impact directly, with hands-on exercises measuring and mitigating unfairness in a model — the ideal companion to this page.