Maximum Likelihood Estimation

The maximum likelihood principle

Section 5.4 defined estimators by guessing a formula and then checking its bias and variance. A learning system needs the reverse: a recipe that produces a good estimator for any model we write down. The most common recipe is the maximum likelihood principle.

The idea is the one a detective uses. Among the explanations the model family allows, prefer the one under which the evidence we saw would have been least surprising.

We have examples drawn independently from the true but unknown distribution . We pick a family indexed by the parameter . The maximum likelihood estimator is the parameter that makes the whole dataset most probable:

The symbol returns the value of , not the value of the function. The expression , viewed as a function of , is the likelihood. Equation 5.57 makes it a product because the examples are independent.

The model is a Gaussian with mean and standard deviation . Each stem shows how probable the model finds one data point, and the likelihood is the product of the stem heights. Slide the mean away from the data and the stems shrink, so the likelihood collapses. Make the standard deviation very small and the stems far from the mean vanish. The lower chart shows the log-likelihood against the mean alone: one smooth peak.

For a Gaussian family the maximum has a closed form, visible in the chart: the best mean is the sample mean, and the best variance is the average squared distance from it. That second quantity is the biased sample variance of Equation 5.36, so maximum likelihood can return an estimator that is biased in small samples. Section 5 of this module explains why it is usually accepted.

Logarithms and the empirical expectation

The product in Equation 5.57 is awkward. Each factor is a probability below 1, so the product shrinks fast and a computer rounds it to zero: this is numerical underflow. The logarithm is strictly increasing, so the value of that maximizes a quantity also maximizes its logarithm, and it turns the product into a sum:

The demo shows both effects. The upper chart rescales the likelihood and the log-likelihood to one range. The lower chart multiplies densities in double precision next to the sum of their logarithms.

With enough examples the naive product reaches the smallest number the computer can represent and becomes exactly 0. From then on every parameter value looks equally terrible, and an optimizer cannot tell them apart. The sum of logarithms keeps going, because it grows only linearly with . That is the practical reason for Equation 5.58. Sums are also easier to differentiate than products, and learning algorithms differentiate.

The empirical expectation

Dividing the sum in Equation 5.58 by does not change the arg max, and it turns the sum into an average. That average is an expectation with respect to the empirical distribution , which puts probability 1/ on each training example:

This is a change of notation. It lets us compare two distributions, the one defined by the data and the one defined by the model.

Two examples you can compute by hand

The closed forms below are added here and not printed in the book.

The peak always sits at the observed fraction of heads. With few flips the likelihood is wide: many coins explain the data about equally well. With many flips it narrows, the same fact as the shrinking standard error of Section 5.4.3. If every flip is a head, the maximum sits at : maximum likelihood would claim that a coin cannot land tails after seeing three heads. Section 5.6 returns to this weakness.

Maximum likelihood as minimizing KL divergence

One way to see what maximum likelihood does is to ask how far the model is from the empirical distribution . The book measures the gap with the Kullback–Leibler (KL) divergence, which compares two distributions by the average log-ratio of their probabilities:

The divergence is never negative, and it is 0 exactly when the two distributions are equal. Its first term involves only the data, so training can ignore it. What remains to minimize over is

which is exactly the maximization of Equation 5.59 with a minus sign.

The book states the equivalence in words. The split of the divergence into a data term and a cross-entropy term is added to the book’s text, and it is in the deeper block below. The demo uses a six-sided die: the data are simulated rolls, and you set six weights that are normalized into probabilities.

Start with the uniform model and reshape it by hand. Each change that moves the bars towards the data lowers the cross-entropy and the KL divergence together. The KL divergence reaches 0 only when the model copies the empirical distribution. The button builds that model, which is a rough sketch with 6 rolls and close to the truth with 1000.

Naming and a warning about sign

Many authors use cross-entropy only for a Bernoulli or softmax negative log-likelihood. The book calls that a misnomer: any negative log-likelihood is a cross-entropy. Minimizing the negative log-likelihood (NLL) is the software name for maximum likelihood. Unlike the KL divergence, the NLL has no floor of zero.

The next demo shows why. A probability density can exceed 1, because it only needs to integrate to 1 over its range, and the log of a density above 1 is positive.

Squeeze the data into a narrow cluster and the best Gaussian becomes tall and narrow. Its peak is far above 1, the average NLL dips below 0, and nothing is wrong. Negative values are legitimate for continuous data, whereas the KL divergence stays at or above zero.

Conditional log-likelihood and mean squared error

Most of supervised learning predicts a target from an input , so the object to estimate is . The conditional maximum likelihood estimator is

For i.i.d. examples the probability factorizes, and after taking logs:

Linear regression as maximum likelihood

Section 5.1.4 chose the weights by minimizing the mean squared error, which the book calls arbitrary. Let the model output a Gaussian for with the prediction as its mean:

The variance is fixed in advance. The log of one Gaussian density is a constant plus a squared error:

The sum over the independent examples gives the conditional log-likelihood of Equation 5.64, which the book writes as

Compare it with the training mean squared error:

The weights appear only in the last term of Equation 5.65, which is times the MSE. The first two terms do not depend on . So maximizing the log-likelihood over the weights is the same as minimizing the training MSE, for any . Drag the points and move the line to see both curves.

Drag the slope slider. The two lower curves have different heights, but both have their lowest point at the same slope, marked by the dashed line. Change and the NLL curve changes in height and sharpness while its lowest point stays still. The least-squares line is therefore the maximum likelihood line under Gaussian noise. The button labeled Best sets to its own maximum likelihood value.

The likelihood chooses the loss

The justification works in both directions. MSE is the right loss if the noise is Gaussian. A different noise model gives a different loss.

The check box Laplace-noise fit shows the best line under a Laplace (double-exponential) noise model, whose negative log-likelihood is proportional to the sum of absolute errors. This extra is not part of the book’s text. Add the outlier and compare the two lines: squared error is pulled hard towards the outlier, while absolute error barely moves.

Properties of maximum likelihood

The book names two asymptotic properties: consistency and efficiency.

Consistency and its two conditions

Under suitable conditions the maximum likelihood estimator is consistent, as defined in Section 5.4.5: with more examples the estimate converges to the true parameter. The conditions are:

• The truth lies inside the model family. The true distribution must belong to the family, or no estimator can recover it.
• The truth corresponds to exactly one parameter value. Otherwise maximum likelihood can recover the right distribution but cannot tell which parameter produced it.

The first demo breaks the first condition. The model is always a Gaussian. The data are either Gaussian, so the truth is inside the family, or an even mix of two well separated Gaussians, so the truth is outside it. After fitting by maximum likelihood, the demo measures the KL divergence from the true distribution to the fitted one, averaged over 40 simulated experiments for each dataset size.

With the truth inside the family, the divergence falls roughly in proportion to 1/m towards 0. With the truth outside, it stops falling and settles at a positive level. Every model in practice is somewhat wrong, so this second case is the normal one.

The second demo breaks the other condition. The model is a Gaussian with unit variance whose mean is the sum of two parameters, . Only the sum affects the data, so many pairs describe the same distribution.

More data make the ridge thinner, so the sum is pinned down more accurately. Six optimizers all reach the maximum, at different parameters. Maximum likelihood recovers the distribution, but the parameters are not identifiable.

Statistical efficiency and the Cramér–Rao bound

Other inductive principles besides maximum likelihood are also consistent, and consistent estimators can still differ in statistical efficiency: one may reach a lower generalization error with a fixed number of samples , or may need fewer examples for a fixed error.

Efficiency is measured by the expected squared error between estimated and true parameter, over training samples. For large the Cramér–Rao lower bound (Rao 1945, Cramér 1946) says that no consistent estimator has a lower mean squared error than maximum likelihood. The final demo estimates the center of a symmetric distribution with the sample mean and the sample median, both consistent and unbiased. The vertical axis is the MSE divided by the bound, so an efficient estimator sits at 1.

On Gaussian data the mean sits on the bound, and the median needs about 57% more data for the same error. The ratio tends to π/2, about 1.57. On Laplace data the roles reverse.

When data are scarce: regularization

For these reasons, consistency and efficiency, maximum likelihood is often the preferred estimator in machine learning. When examples are few and overfitting appears, the book says that regularization strategies such as weight decay may be used to obtain a biased version of maximum likelihood with less variance.

The last demo estimates six regression weights from few noisy examples. For each penalty strength it repeats the experiment 300 times and splits the squared error of the weight estimate into bias and variance. A penalty of 0 is plain maximum likelihood.

• The maximum likelihood estimator maximizes the probability of the data (Eqs. 5.56 and 5.57), and logs avoid underflow (Eq. 5.58).
• It is the same as minimizing the KL divergence, the cross-entropy or the NLL (Eqs. 5.60 and 5.61). A Gaussian with fixed variance turns the conditional version into the MSE (Eqs. 5.65 and 5.66).
• It is consistent and efficient for large , provided the truth lies in the family.
• With few examples, weight decay gives a biased estimate with less variance.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 5.5 (pages 131–135), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.