Learned Approximate Inference

Wake-sleep

Chapter 19 has so far found by running an optimization for each input . That can be slow. The idea of learned approximate inference is to treat the optimization as a function that maps to an approximate posterior, and to replace the function with a neural network. The network is then an inference network, and one pass through it gives an answer.

The difficulty is that there is no supervised data for it. For a given , nobody knows the correct , and the right answer changes as the model changes during learning. The wake-sleep algorithm solves this by sampling both and from the model. In a directed model, ancestral sampling does this cheaply: start at , then produce . The inference network is then trained to reverse the mapping, predicting which caused the sampled .

The book notes that this has a drawback. The inference network only learns on values of that the model produces with high probability. Early in learning the model does not resemble the data, so the network has little chance to learn from data-like inputs.

The book also relates the schedule to sleep. Monte Carlo training of undirected models struggles when the positive and negative phases run in long separate blocks. A bound-based method can run long periods of improving and long periods of improving , which is closer to how animals alternate waking and sleeping. The book is clear that this is speculation.

Training an inference network from samples

The demo uses the smallest possible case, where everything is Gaussian. This example is our own illustration, not one from the book, which gives no numbers for it. The model has and . In the sleep phase we draw pairs from this model and fit a linear network by least squares. For a linear Gaussian model, the best linear predictor of from is exactly the posterior mean. So the learned slope equals the exact slope of the posterior mean:

The data is not always generated by the model. The demo lets the data be generated with a different scale , so that . The exact posterior slope for that data is , and the network still uses its fixed slope . Move the slider and watch the two lines separate.

The second challenge shows the other side of the error. For a steep posterior the fixed slope can be far too small, and the network under predicts the hidden value.

Other learned inference methods

The same strategy has been applied to other models. Salakhutdinov and Larochelle (2010) showed that one pass through a learned network can be faster than iterating the mean field fixed point equations in a deep Boltzmann machine. Their training runs the network, applies one step of mean field to refine its output, and trains the network to output that refined estimate.

Predictive sparse decomposition, from section 14.8, trains a shallow encoder to predict a sparse code for the input. It is a hybrid of an autoencoder and sparse coding, and its encoder can be read as learned approximate MAP inference. A shallow encoder cannot capture the competition between units that mean field produces, which is explaining away from section 19.4. A deep encoder can, and the ISTA method of Gregor and LeCun (2010) uses one.

The demo in section 2 shows the limit of this idea. Each method learns from samples or from a fixed objective, and the answer is only as good as the training distribution covers. A learned encoder that works well on one kind of input can fail on another.

The overlap is the area under the lower of two densities. It is one when the sleep samples and the data cover the same values of , and it falls as their spreads differ.

Pointer to the variational autoencoder

Learned approximate inference has become one of the main approaches to generative modeling in the form of the variational autoencoder (Kingma, 2013; Rezende et al., 2014). The method needs no explicit targets for the inference network. The network defines the bound directly, and its parameters are adapted to increase the bound, so no sampling phase is needed.

The book describes this model in depth in section 20.10.3, which is the next place to read for the details.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 19.5 (pages 651–653), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.