Other Generation Schemes

Destroying structure with diffusion

The methods of this chapter draw samples with MCMC, with ancestral sampling, or with a mixture of the two. The book says these are the most popular approaches and by no means the only ones. This page covers two more: a model built by reversing a diffusion, and sampling by approximate Bayesian computation.

Think of a drop of ink in water. The ink spreads until the water has an even colour. The spreading destroys the structure of the drop and raises the entropy, a measure of how spread out a distribution is. Sohl-Dickstein et al. (2015) turn this around for learning. Start with data that have structure, destroy the structure with a diffusion that gradually adds noise, and train a model that runs the process in reverse. The model removes the noise one small step at a time, until it reaches the data.

The book describes the forward process in words: it gradually changes the distribution to have more entropy. The toy below makes this exact. Each bump of the data has spread , and the added noise has variance . The bumps keep their centres, and their spread grows to (added here, not in the book):

The forward process is a sequence of small Gaussian steps, each with noise size . Gaussian steps add their variances, so in the toy the mixture above is exactly what the whole sequence gives. The entropy of the mixture is defined as an integral, and the demo computes it numerically:

The data entropy is the readout at level 0. The entropy rises with the diffusion level, and it rises most quickly at first. Once the bumps merge, the density is close to a single broad Gaussian, and its entropy keeps growing only slowly. The book's claim is the first half of this: diffusion moves the data towards a distribution with more entropy, and the reverse process then has to undo it.

Undoing one step of diffusion

Diffusion inversion trains a transition operator that undoes the effect of the diffusion. The book notes that this is very close to the generative reading of the denoising autoencoder in section 20.11.1. Both learn to probabilistically undo the effect of adding noise. The difference is that diffusion inversion needs to undo only one step of the diffusion, rather than travelling all the way back to a clean data point.

The model is the probability distribution produced by the final step of the chain, so the many steps add no approximation of their own. The approach resembles MCMC in using many iterations to produce one sample. The book's contrast with MCMC is that the model is defined by the final step, so no approximation comes from the iterations.

The book names the dilemma that this solves. With small noise, a denoising autoencoder sees only configurations near the data points. With large noise, it is asked to do an almost impossible job, because the denoising distribution is highly complex and multi-modal. Undoing one step addresses this dilemma. The learner can learn the shape of the density around the data points more precisely, and it can remove spurious modes that show up far from the data.

The toy takes the data density as the earlier state and adds one step of Gaussian noise with size . The one-step reverse distribution then follows from Bayes' rule. Its form is a mixture of two Gaussians, one per data bump, and the book's dilemma can be checked directly on it. The equations are added here and are not in the book:

With a small one-step noise, the reverse distribution has one bump, close to the current state. This is the easy case, where the learner sees configurations near the data. With a larger noise the two candidate bumps can both survive, and the reverse distribution becomes multi-modal. In this toy, at the midpoint, that switch happens at a noise size of about 0.13. Move the noise through that value and watch the bump count.

Approximate Bayesian computation

The second scheme is approximate Bayesian computation, or ABC, the framework of Rubin et al. (1984). Samples are rejected or modified so that the moments of selected functions of the samples match the moments of the target distribution. The book notes that this uses the moments of the samples, like moment matching, but it is different. Moment matching trains the model to emit samples with the right moments. ABC changes the samples themselves.

Bachman and Precup (2015) showed how to use ideas from ABC in deep learning, by using ABC to shape the MCMC trajectories of GSNs. The toy below shows the basic rule. Draw candidates from a broad Gaussian with spread 3. Keep a candidate only if its summary is within of the target moment , which is the second moment of the data, 4 plus 0.25 (added here). The data have and , computed from the two bumps:

A small tolerance keeps few candidates, and those cluster near the two values where . For small tolerances the fourth moment of the kept samples stays near 18, far from 22.19. A large tolerance keeps more candidates, but they spread out and pull their second moment away from the target. Changing the tolerance cannot fix the fourth moment, because the rule constrains one summary of the kept samples, not the whole distribution.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.13 (pages 716–717), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.