Stochastic Encoders and Decoders

The decoder is a distribution

Autoencoders are feedforward networks, so the loss functions and output units used for ordinary networks also work here. Think of a translator who offers a range of likely sentences instead of one. The loss rewards the sentence that matches the original.

Given a hidden code , the decoder gives a conditional distribution ( given ). Training minimizes the negative log likelihood = −log . The output unit depends on the form of the distribution. For real-valued , a linear output unit gives the mean of a Gaussian, and the negative log likelihood becomes the mean squared error. The formula below is the Gaussian case for one input value, added here to show where the squared error comes from, and is not printed in the book:

Here is the target, is the mean the decoder outputs, and is the spread of the output noise. The squared error sits in the top of the fraction, so a smaller makes the loss steeper.

The demo shows the loss against the mean. Its lowest point is always at the target, whatever the spread. The spread only changes how steep the sides are, and so how strongly the loss punishes a miss.

Binary and discrete outputs

When the input is binary, the book uses a Bernoulli distribution whose parameter comes from a sigmoid output unit. The output is a probability that the bit is 1, and the loss is the negative log of the probability given to the true bit. For discrete values, a softmax gives a distribution over the possible values. The book notes that the output variables are usually treated as conditionally independent given , so the probability is cheap to evaluate. Mixture density outputs can model correlated outputs when that is needed.

The formula below uses a logit , the input to the sigmoid. It is added here for the demo, and is not printed in the book:

The loss is smallest when the output probability matches the target bit. A confident wrong answer costs much more than an uncertain one, because the loss grows without bound as the probability of the true bit goes to zero.

Stochastic encoders

The encoder does not have to be a function. The book generalizes the encoding function to an encoding distribution ( given ), as in figure 14.2. Any latent variable model defines both a stochastic encoder and a stochastic decoder, with the two equations below:

The book warns that the encoder and the decoder are not necessarily conditionals of one joint distribution. Training them as a denoising autoencoder tends to make them compatible, as section 4 explains. The demo below takes the simplest stochastic encoder, a Gaussian with a fixed mean. The formula is added here for the demo and is not printed in the book:

The same input now gives different codes on different runs. The demo fixes the random seed, so you see the same 200 draws each time, scaled by the spread you choose.

A stochastic autoencoder

Figure 14.2 shows the structure: noise can enter the encoder, the decoder, or both. Switch the noise on for each part below, and watch which parts of the machine are sampled from a distribution.

The book cites Alain et al. (2015): training the encoder and the decoder as a denoising autoencoder tends to make them compatible, asymptotically, with enough capacity and examples. The demo shows which parts are random, and not whether the two parts agree.

• A decoder can be a distribution , and its output unit follows the noise model. A Gaussian gives the squared error, and a sigmoid gives the Bernoulli loss.
• A stochastic encoder is a latent variable model that samples the code, as equation (14.12) shows.
• Encoder and decoder need not be conditionals of one joint model, and denoising training tends to make them compatible.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.4 (pages 509–510), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.