Evaluating Generative Models

Scores that move with the units

Evaluating a generative model is harder than it looks. The usual score is the log-likelihood, the average over test data of the log of the density that the model gives each point. Section 20.14 of the book lists several ways that a reported number can mislead. The first is the comparison itself. A stochastic estimate, a number that changes from run to run, cannot be ranked against a deterministic lower bound, a fixed number that the true value is never below, unless you know how loose the bound is. If the aim is a practical task, such as anomaly detection, which flags unusual examples, a model can be preferred on a criterion that fits the task, such as the precision and recall of its ranking of test examples.

The estimator can also mislead. Many models define a density only up to a constant. The model gives an unnormalized score , and the density is that score divided by the partition function , the total mass that makes the density add up to one. The book's example uses annealed importance sampling (AIS) to estimate . A cheap AIS run may miss some modes of the model and underestimate , which overestimates . So a high score can come from a poor estimator rather than a good model.

Preprocessing is the next issue. In object recognition, slightly different preprocessing for each algorithm is usually acceptable. In generative modelling it is not, because any change to the input changes the distribution to be captured. The book's example is that multiplying the input by 0.1 artificially increases the likelihood by a factor of 10. The reason is the change of variables for a density. If the data are stretched by c, the density must shrink by the same factor, or it would no longer add up to one, and that gives the term minus log c. The equation below is added here and is not in the book:

Scaling the data by changes the log-likelihood by , so a small raises the score with no change in the model. The toy below fits a Gaussian to the scaled data each time, and evaluates it on the same scaled data. The model family is fixed, so the only change is the units.

The curve is a straight line in the natural log of the scale, with slope . Going from the original units to a tenth of them raises the score by ln 10, about 2.3 nats. The plot uses log10 of the scale, so there the same line has slope about −2.3. The score measures the units as much as the model, which is the book's warning. Preprocessing has to be the same for every model that is compared.

Binarization changes the number

The book notes that MNIST images are grayscale, and that models treat them in three ways. Some treat them as points in a real vector space, some as binary, and some treat the grayscale values as probabilities for binary samples. Real-valued models should be compared only to other real-valued models, and binary models only to other binary models. For binary models, the log-likelihood is at most zero, while for real-valued models it can be arbitrarily high, because it measures a density.

Among binary models the binarization has to match too. Binarization turns each grey pixel into a 0 or 1. A grey pixel can be thresholded at 0.5, or it can be drawn at random with probability equal to its intensity. Random binarization also has variants. The book names three: binarize the dataset once, draw a new bit for each example at each training step, or draw several binary samples for evaluation. The book says each of these schemes gives wildly different likelihoods, so training and evaluation must use the same one. This is why researchers often share a single file of one random binarization.

The toy takes one pixel with grey level . Its best model is exact in each case, and the two average log-likelihoods are below. The equations are added here and are not in the book:

The two scores measure different things. The thresholded model reports a perfect zero because the bit it models is fixed. The random model reports the entropy of the bit, which is largest at grey level 0.5. Neither number says which binarization is better, so a comparison must fix the scheme first.

A likelihood that rewards constant pixels

The book says that sample quality is not a reliable guide, because a very poor probabilistic model can produce very good samples. Likelihood can help, but it has its own trap. Real-valued models of MNIST can reach arbitrarily high likelihood by assigning arbitrarily low variance to background pixels that never change. The book says that this potential exists for any maximum likelihood problem with real values, and that it is especially problematic for MNIST, where so many output values are trivial to predict.

The toy has two pixels. The stroke pixel varies, and its best Gaussian fit has spread 0.1 and average log-likelihood about 0.88. The background pixel is always 0, and its model has spread . The equations are added here and are not in the book:

The stroke pixel contributes the same amount at every setting. The background pixel contributes a term that grows without limit as its spread falls, so the total score rises without limit. A model that has learned to predict a constant pixel gains nothing useful, but the likelihood cannot tell this apart from a model that has learned real structure. This is why the book calls for other ways of evaluating generative models.

Forward and reverse KL

The Kullback-Leibler (KL) divergence measures how far one distribution is from another. It is zero only when the two agree, and it is not symmetric in its two arguments.

Theis et al. (2015) review many issues in evaluating generative models. They stress that the metric must match the intended use. Some models assign high probability to most realistic points, while others rarely assign high probability to unrealistic points. The difference can come from whether the model minimizes the KL divergence in one direction or the other, as illustrated in the book's Figure 3.6. This page does not reproduce that figure, and uses a toy instead.

The two directions are and . The first penalizes the model for leaving data uncovered. The second penalizes the model for placing mass where the data are absent. The equations are the standard definitions, added here and not written out in the book:

The forward KL is small only when the model puts mass on both bumps, which needs a wide Gaussian and so a poor fit to either. The reverse KL is small when the model sits on one bump and places no mass in the gap, which is why the reverse direction is called mode seeking: it locks onto one mode of the data and leaves the others out. In this toy the best reverse fit sits on one bump, with KL of log 2, about 0.69 nats. The best forward fit is wide and centred in the gap, with KL about 0.72 nats. One Gaussian cannot match two bumps, so each direction picks a different compromise, and the choice of direction decides which failure is cheaper.

Copies, missing modes and nearest neighbours

Sample quality can hide two failures. A model can copy training examples, which looks perfect and is not a model of the distribution. Or it can miss some modes, which a human observer would not easily see in a large set of samples. The book gives a dog-and-cat example: a model that reproduces only the training dogs has overfit, because it makes no new images, and has underfit, because it assigns no probability to the cats. Each dog looks good on its own. Researchers often judge samples by eye, and the best practice is to have subjects who do not know where the samples came from (Denton et al., 2015). With tens of thousands of modes, a model can ignore a few, and no observer can inspect enough images to notice the gap.

A common check is the nearest-neighbour test, illustrated in the book's Figure 16.1. The idea is to show generated samples next to their nearest training example, in Euclidean distance. A sample that is very close to a training example is a copy. The toy below uses 40 training points from the two bumps and three models. A copy model reproduces the training points with a small added noise, a model with only the right bump misses the left bump, and a model with both bumps covers the data.

The copy model covers both bumps, so the coverage check passes it. Its nearest-neighbour distance is set by the copy noise. With small noise the median distance is near zero, and the distance check flags the copy. With larger noise the copies blur, and their distance moves towards that of a model that covers both bumps without copying. The right-only model shows the other failure: its distances are about as small as those of the both-bumps model, yet the left bump is missing. So the distance check alone cannot tell a missing mode from a good model, and the coverage check is needed for that.

The book's conclusion for the whole section is that no current metric is free of serious weaknesses, and that designing better measures is one of the main research topics in generative modelling. The toy is a small case of that problem: each check reports on one side of the trade-off.

• A score can move with the units. Multiplying the data by a constant changes the log-likelihood by minus the log of that constant, with no change in the model, so preprocessing must match across models.
• Binary and real-valued likelihoods are not comparable, and different binarizations give different numbers. Compare only models that share the same space and the same binarization.
• A real-valued likelihood can rise without bound by shrinking the variance of a constant pixel. Sample quality alone does not reveal this, and neither does the likelihood alone.
• The forward KL and the reverse KL reward different things. The forward direction penalizes uncovered data, and the reverse direction penalizes mass where data are absent.
• The nearest-neighbour test flags copies, and a coverage check flags missing modes. A model can pass one and fail the other, so both are needed.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.14 (pages 717–720), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.