Generative Stochastic Networks

The latent state in the chain

Section 20.11 ran a chain that moves only the visible variables. The generative stochastic network, or GSN, adds latent variables to the chain, next to the visible variables . The book describes a GSN as a generalization of a denoising autoencoder. It keeps the same idea: the model is specified by the steps of the chain, and not by a written-out joint distribution.

Two conditional distributions specify one step of the chain. The book gives them as follows.

1. tells how to generate the next visible variable given the current latent state. This is the reconstruction distribution, which also appears in denoising autoencoders, RBMs, DBNs and DBMs.
2. tells how to update the latent state, given the previous latent state and the previous visible variable.

A GSN does not write down the joint distribution of the visible and latent variables. The joint is defined implicitly, if it exists, as the stationary distribution of the generative chain. The conditions for a stationary distribution are the standard MCMC conditions. They can fail for some choices of the transition distributions, for example when the transitions are deterministic.

The traces start both states at 3. Move the number of steps and read the readout, which splits each new value into its weighted part and its noise. The latent state is never shown in the data. The chain carries it from one step to the next, so the visible state depends on the past through . The toy uses linear maps with Gaussian noise, which the book does not require. The equation below is not in the book:

Two conditionals and their stationary spread

Stack the pair and into one vector. Then one step of the toy is a matrix applied to the previous pair, plus noise with covariance . The matrix has rank one, so one eigenvalue is 0 and the other is . The spectral radius therefore decides whether the chain settles:

When is below 1, the covariance of the pair settles to a fixed value, the stationary covariance. Its equation is written below, and the variance of the visible state is the (2,2) entry. When is 1 or more, no fixed value exists.

The demo settles the covariance by repeating the update, which converges when is below 1. It then checks the answer against 2000 simulated chains.

Press Oscillation case to set a = −1, b = 0, c = 1 and = 0. Here = 1. The state flips sign at every step and never settles, which is the boundary of the rule. A value of above 1 is a different failure: the state grows, and the chain has no stationary distribution at all. Raise the noise and the spread grows without limit, because at this boundary no stationary covariance exists.

Reconstruction log-probability criterion

The training criterion that Bengio et al. (2014) proposed and evaluated is the reconstruction log-probability on the visible units, as for denoising autoencoders. The chain is clamped at a training example, . The model then maximizes the log-probability that the visible unit generates at a later step , where the latent state is sampled from the chain given the clamped start. The book says the gradient is estimated with the reparametrization trick of section 20.9, and that the walk-back procedure of section 20.11.3 is used to improve convergence.

The book gives the criterion in words, and it does not write out the toy's value. The equation below is a toy form, in which the reconstruction distribution is Gaussian with the same noise size as the chain. The average is taken over the chains at each step:

At step 1 the chain has moved only a little from the example, so the latent state still reflects it and the log-probability is high. As the steps go on, the chain forgets the example and the curve moves towards its stationary value. When is above 1 the curve has no such value, and it falls without limit. That is the same failure as in section 2, now seen through the training criterion. The book says the training criterion is about the visible units, so the demo measures the reconstruction of and nothing else.

Discriminant GSNs

The original GSN was meant for unsupervised learning, and it models for observed data . The framework can also model a conditional . Zhou and Troyanskaya (2014) generalize GSNs in this way. They back-propagate the reconstruction log-probability over the output variables only and keep the inputs fixed. They applied it to protein secondary structure, with a one-dimensional convolutional structure in the transition operator.

The Markov chain is then over the output, and the input sequence only conditions it. The book calls this a case of structured outputs. Zöhrer and Pernkopf (2014) go a step further and combine the two objectives. They add a supervised cost and an unsupervised cost with different weights, which are the reconstruction log-probabilities of y and of x. Larochelle and Bengio (2008) had introduced this hybrid criterion for RBMs, and Zöhrer and Pernkopf report improved classification with it.

The toy below uses two quadratic costs, which are not in the book, to show how the weight moves the best parameter. The supervised cost prefers and the unsupervised cost prefers . The total cost is their weighted sum:

The total cost is lowest at , which is in the toy. A small weight on the supervised cost pulls the best value towards 0, and a large weight pulls it towards 2. The book does not give this toy, so the numbers come from the formula above and not from a trained model.

• A GSN is a Markov chain with two conditionals, the reconstruction distribution and the latent update . Its joint distribution is implicit, as the stationary distribution of the chain.
• In the linear toy the chain settles when is below 1. At 1 or above, the spread has no fixed value.
• Training maximizes the reconstruction log-probability of a clamped example at step . Early steps stay near the example, and the average moves towards its stationary value as the chain forgets it.
• Discriminant GSNs back-propagate the reconstruction log-probability over the outputs only. The hybrid weight, in the toy, sets where the best parameter sits.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.12 (pages 714–716), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.