A bank's model denies someone a loan. "Why?" they ask — and it is not an idle question: in many jurisdictions the bank is legally required to answer. But the model is a gradient-boosted ensemble of a thousand trees, a black box that maps inputs to a number with no sentence attached. The model was optimised to be accurate, not explainable, and those are different goals that frequently pull apart. Interpretability is the discipline of prying a human-readable "why" out of a machine that was never built to give one.
This matters for far more than compliance. You cannot trust a model you cannot understand, you cannot debug one whose mistakes you cannot trace, and you cannot check it for fairness if you do not know what it keys on.
Every interpretability method answers one of two different questions — and confusing them is a classic error.
A feature can be globally unimportant yet decisive for one unusual case, or globally important yet irrelevant to a particular person. You need both altitudes, and you must be clear which one a given tool gives you.
The simplest global method is delightfully direct. Take a feature, shuffle its column so its values are randomly reassigned across rows — destroying any relationship it had with the target while leaving every other feature intact — then re-score the model. If accuracy collapses, the model leaned heavily on that feature; if the score barely moves, the feature was doing little work. Rank all features by how much scrambling each one hurts, and you have permutation importance.
It is model-agnostic (works on any predictor), intuitive, and honest about what the model
actually uses rather than what it was told about — closely related to the
For a local "why me?", the most principled tool borrows an idea from cooperative game theory. Imagine the features are players cooperating to produce a prediction, and the payout is how far that prediction sits above the average. How do you split the credit fairly among the features? The economist Lloyd Shapley solved exactly this in 1953: the Shapley value of a feature is its average marginal contribution across every possible order in which features could be added to the team.
So for the denied applicant, SHAP might report: base rate 40% approval, −25 points for the short credit history, −10 for the high requested amount, +5 for stable employment — a clear, additive story that sums to the final score.
A partial dependence plot answers a third question: as one feature sweeps across its range — holding the others fixed on average — how does the model's prediction move? Sweep "income" from low to high and plot the average predicted approval, and you see the shape of the relationship the model learned: linear, flat then rising, or a non-monotone hump. It reveals not just which features matter (permutation importance) but how — the direction and curvature of each effect.
Sometimes you should — a well-regularised linear model or a shallow decision tree is inherently interpretable, and on tabular data the accuracy gap to a black box is often smaller than people assume. The researcher Cynthia Rudin argues forcefully that for high-stakes decisions we should prefer models that are interpretable by design rather than opaque models patched with post-hoc explanations, because those explanations can themselves be unfaithful to what the model really does. The counterweight is that on images, text and audio the accuracy gap is enormous, and there a black box plus honest interpretability tooling is the pragmatic answer. The choice is a genuine trade-off, not a solved question.
The deepest trap: interpretability tells you what the model relies on, not what is true of the world. If SHAP says a model heavily uses "postcode," that is a fact about the model's wiring — it does not mean postcode causes the outcome, and acting on it as if it did repeats every confounding error. Worse, two correlated features can trade importance almost arbitrarily: shuffle one and the model leans on its correlated twin, so permutation importance can under-credit a feature whose information is duplicated elsewhere. Read every explanation as "this model, on this data, keys on this" — a description of the predictor, never a licence to infer cause.
Kaggle Learn's Machine Learning Explainability course walks through permutation importance, partial dependence plots and SHAP values hands-on in Python — computing exactly the explanations described here on a real trained model.