Likelihood and MLE

You flip a coin 10 times and get 7 heads. Is the coin fair? If not, what is its true bias — the probability \theta it lands heads? You can't see \theta directly; all you have is the data. So ask the question that founds nearly all of statistical estimation: which value of \theta makes the data I actually observed the most probable?

That is maximum likelihood estimation (MLE). It doesn't guess and it doesn't need a prior belief — it simply lets the data speak, and reports the parameter under which the observed outcome is least surprising. For the 7-heads coin, the answer will turn out to be exactly what your gut hoped: \hat\theta = 0.7.

The likelihood, and the estimate that maximises it

Flip the likelihood around. Fix the data and let the parameter vary: the likelihood function L(\theta) = P(\text{data} \mid \theta) scores how well each candidate \theta explains what we saw. The maximum-likelihood estimate (MLE) is the parameter that scores highest:

\hat\theta_{\text{MLE}} = \arg\max_{\theta} \, L(\theta) = \arg\max_{\theta}\,\prod_i P(x_i \mid \theta).

Because products of many small probabilities are awkward, we almost always maximise the log-likelihood \ell(\theta) = \sum_i \log P(x_i\mid\theta) instead — the logarithm turns the product into a sum and does not move the maximum.

Worked example: the biased coin

Back to the coin: 7 heads in 10 flips, with unknown bias \theta. Each flip is independent, so the likelihood of this exact record is

L(\theta) = \theta^{7}\,(1-\theta)^{3}.

To find the peak, take the log (the product becomes a sum) and differentiate:

\ell(\theta) = 7\log\theta + 3\log(1-\theta), \qquad \ell'(\theta) = \frac{7}{\theta} - \frac{3}{1-\theta}.

Set \ell'(\theta)=0: then 7(1-\theta) = 3\theta, so 7 = 10\theta and

\hat\theta = \frac{7}{10} = 0.7.

The MLE of a coin's bias is simply the observed proportion of heads. Nothing exotic — but notice the log did the heavy lifting: differentiating \theta^7(1-\theta)^3 directly would have meant a messy product rule, while the log split it into two clean terms.

Worked example: estimating a rate from counts

A café counts customers arriving each hour over four hours: 2, 5, 3, 6. Model each count as \text{Poisson}(\lambda), where \lambda is the unknown average rate. The log-likelihood of n counts x_i is

\ell(\lambda) = \sum_i \big(x_i\log\lambda - \lambda\big) + \text{const}, \qquad \ell'(\lambda) = \frac{\sum_i x_i}{\lambda} - n.

Setting \ell'(\lambda)=0 gives \hat\lambda = \tfrac{1}{n}\sum_i x_i — again the sample average. Here that is (2+5+3+6)/4 = 4 customers per hour. A recurring theme: for many everyday models, "maximise the likelihood" quietly reduces to "take the average".

Gaussian noise makes MLE into least squares

Suppose each measurement is the truth plus independent Gaussian noise, x_i = \theta + \varepsilon_i with \varepsilon_i \sim N(0, \sigma^2). The log-likelihood is

\ell(\theta) = -\frac{1}{2\sigma^2}\sum_i (x_i - \theta)^2 + \text{const}.

Maximising \ell is the same as minimising the sum of squared residuals \sum_i(x_i - \theta)^2. That is the deep reason least squares is everywhere: least squares is maximum likelihood under Gaussian noise. For estimating a single mean, the MLE is just the sample average \hat\theta = \bar x.

The likelihood peaks at the best fit

Three measurements with sample average \bar x = 3. The curve is the likelihood of the mean \theta; it is a Gaussian centred on \bar x — the MLE. Shrink the noise \sigma and the peak sharpens: less noise means the data pins the estimate down more tightly.

The likelihood L(\theta) = \prod_i P(x_i\mid\theta) scores parameters by how well they explain the data; the MLE maximises it.
Maximise the log-likelihood (a sum) for convenience — same maximiser.
Under independent Gaussian noise, MLE = least squares; for a single mean, the MLE is the sample average.

The single deepest confusion in this whole topic: L(\theta) = P(\text{data}\mid\theta) is the probability of the data, viewed as a function of \theta. It is not a probability distribution over \theta. Two consequences that trip people up:

It does not integrate to 1. Sweep \theta across its range and the area under L(\theta) can be anything — the likelihood makes no promise to normalise, because it isn't a density in \theta.
"Likelihood of the parameter" ≠ "probability of the parameter." Saying "there's a 70% chance the coin's bias is 0.7" is a statement about P(\theta\mid\text{data}) — the posterior — and getting there needs a prior and Bayes' theorem, which is exactly what MAP estimation adds on top of MLE.

Maximum likelihood was pioneered by Ronald Fisher in the 1920s, and it turns out to be lurking behind an astonishing range of methods you meet elsewhere under different names. Fitting a line by least squares? That is secretly MLE with Gaussian noise. Logistic regression, and the training of a huge swathe of machine-learning models, all reduce to the same instruction: maximise the likelihood.

When a neural network minimises "cross-entropy loss", it is maximising a log-likelihood in disguise. Fisher's idea is the mathematical form of the oldest wish in science — let the data speak for itself — and a century on, it quietly runs a great deal of the AI around you.