Drawing Samples from Autoencoders

Autoencoders that can already sample

Chapter 14 showed that several kinds of autoencoder learn the data distribution, and the chapter's earlier sections showed how to train generative networks. This section asks a practical question: once an autoencoder has learned, how do you draw a new sample from it?

Think of a photocopier that has learned what good handwriting looks like. Feed it a smudged page and it cleans the page up. A new page of handwriting can then be made by starting from a smudge, cleaning it, smudging the result a little, and cleaning again. The repeated cleaning wanders across the space of handwriting, and each visit to a clean page is a sample.

Different autoencoders answer the question in different ways. The variational autoencoder represents a probability distribution explicitly, so it can use ancestral sampling: draw the top variables first, then each lower variable given the ones above. One pass gives one sample. Most other autoencoders have no such explicit distribution. They need Markov chain Monte Carlo (MCMC) sampling, which runs a Markov chain, a sequence of states where each state depends only on the one before, until its long-run behaviour produces samples from the target distribution.

Contractive autoencoders are designed to recover an estimate of the tangent plane of the data manifold, the thin curved surface on which the data concentrate. Repeated encoding and decoding with some injected noise then makes a random walk along that surface (Rifai et al., 2012; Mesnil et al., 2012). The book calls this manifold diffusion, and it is a Markov chain too. The rest of this page uses a more general chain, one that works for any denoising autoencoder.

The Markov chain of a denoising autoencoder

The book gives one step of the chain in four sub-steps. Start from the previous state . The book's Figure 20.11 draws the same loop as a diagram, and the list below gives the steps in its order:

1. Corrupt. Inject noise. Draw from the corruption process , which turns the state into a noisy copy.
2. Encode. Compute the code from the noisy copy.
3. Decode. Compute the parameters of the reconstruction distribution .
4. Sample. Draw the next state from that reconstruction distribution.

The equations are the book's four sub-steps written in one block. They are not numbered in the book. The encoder and decoder are deterministic. The two random steps are the corruption and the final draw from . The book says that the corruption is a hyperparameter with two roles. It controls how fast the chain mixes, and it controls how much the estimator smooths the empirical distribution.

With squared error, the decoder's output is the mean of a Gaussian, and estimates . The reconstruction distribution then adds Gaussian noise of size around . The book says this second noise level should match the mean squared error of the reconstructions.

To run the chain on numbers, this page uses a toy where the encoder and decoder are exact. The data are one visible number, drawn from two bumps at and , each with spread . The corruption is Gaussian, so the corrupted state is

For this toy the encoder can compute the probability that a corrupted point came from the right bump. That probability is the code. The decoder's mean is the posterior mean of the clean state given the corrupted one. Both are closed forms that the book does not print, so they are derived here:

Here is the shrink factor of the reconstruction, defined as

It is near 1 when the corruption is small and near 0 when it is large. The mean moves towards the bump that the corrupted point most likely came from. The closed forms agree with a numerical integration of the same model.

Press Next sub-step and read the readout after each press. The corrupted point sits near the chain state, but with a shift of . The encoder then reports how likely each bump is. The decoder moves its mean towards the bump that the corrupted point most likely came from. The last sub-step draws around that mean. The next press of the button moves the chain to , and the cycle repeats.

The demo above shows one step. The question that matters for sampling is what happens after many steps. Predict before you run the long chains below.

All chains start in the left bump, at . A chain that stays there has not mixed. A chain that crosses to the right bump has mixed. The total variation distance compares the histogram with the data density, and it is 0 for a perfect match. Two things trade against each other. A small corruption noise leaves the chain stuck. A large one lets it cross, but it spreads the histogram into the gap between the bumps. The reconstruction noise also matters, because a value far from the error gives a histogram that is too narrow or too wide.

Clamping and conditional sampling

Clamping is holding some units at fixed values while the others move. The book names the two sets. The observed units are clamped to their values. The free units are resampled at each step, given the clamped values. The chain then samples the conditional distribution .

Boltzmann machines can do the same, and the book notes that MP-DBMs can be read as a form of denoising autoencoder that samples missing inputs. Alain et al. (2015) pointed out that the transition operator must also satisfy detailed balance, which the book states in words: at equilibrium, the flow from one state to another equals the flow back. The formula below is not printed in the book.

Detailed balance is what keeps the conditional chain in equilibrium in both directions. The toy below has two units. The free unit and the observed unit come from two clusters, at and , each with spread . If you know , that tells you which cluster comes from, so the conditional is a single bump or a mixture of the two:

The weight is the logistic function of , which follows from the separation of the clusters and their spread. (The book does not print this weight.) When is 0 the two bumps are equally likely. When is at least about 0.2 the right bump has at least 95% of the weight.

Move the clamped unit from to . The exact weight of the right bump moves from almost 0 to almost 1, and the histogram of the chain follows it. The chain never sees the clamped unit as a free variable, because it is held fixed in every step. The histogram then approximates the conditional, and the gap to the exact density is the total variation distance .

Walk-back training procedure

The walk-back procedure trains a denoising autoencoder with a walk of several steps instead of one. It starts the chain at a training example, as contrastive divergence does (section 18.2), and it penalizes the last probabilistic reconstruction, or all of the reconstructions along the way. The book gives two claims. Training with steps reaches the same stationary distribution as training with one step. In practice, though, the longer walk removes spurious modes more efficiently. A spurious mode is a bump in the model's distribution where the training data have no mass.

Why would a longer walk help? A walk of one step from a training example barely leaves the example, so the penalty sees only reconstructions close to the data. A longer walk wanders further, and the penalty then acts on the places it reaches. The demo below shows where walks go in the toy, for walks of different lengths.

The lower plot counts walks that end on the other side of zero from their training example. The upper plot shows where the endpoints of the chosen walk length lie. Short walks rarely cross. Longer walks cross more often, so the penalty of walk-back training reaches places that a single reconstruction rarely visits. The demo runs the exact denoiser of the toy, not a trained network, so it shows where walks go and not what a trained model learns.

• A denoising autoencoder defines a Markov chain of four sub-steps: corrupt, encode, decode, sample. Its long-run distribution approximates the data distribution, under the book's consistency condition (Bengio et al., 2014).
• The corruption noise controls how fast the chain mixes and how much it smooths. Small values leave the chain stuck in one bump. Large values mix, but they blur the distribution.
• The reconstruction noise should match the mean squared error of the reconstructions. Too small or too large a value gives a distorted histogram.
• Clamping some units and resampling the others samples a conditional distribution, provided the transition operator satisfies detailed balance (Alain et al., 2015).
• Walk-back training starts its walks at training examples and penalizes the reconstructions they reach. Longer walks reach regions where a one-step penalty rarely acts, which is how the book says spurious modes are removed more efficiently.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.11 (pages 711–714), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.