MathalyseResearch

Learn AI · Chapter 20 · Deep Generative Models

Boltzmann Machines for Structured or Sequential Outputs

Explain how a Boltzmann machine becomes a conditional model for structured outputs and for sequences, how the conditional RBM of Taylor et al. makes its visible bias depend on the past, and how the RNN-RBM lets a recurrent network emit every RBM parameter and is trained through contrastive divergence and back-propagation through time.

Book §20.7, pages 685–686About 30 minutes
Section 1

Conditioning on structured outputs

Some tasks map an input to an output whose entries depend on each other and must obey constraints. In speech synthesis, the output is a waveform, and the whole waveform must sound like one coherent utterance. A natural way to represent those relationships is a distribution over the output given the input. Boltzmann machines, extended to model conditional distributions, can supply that probabilistic model.

the visible vector now the vector one step back RBM weights
Predict first

Why does a structured output task need a joint model over the entries of the output rather than one prediction per entry?

How to use this page

Click any colored symbol in an equation or a sentence and it explains itself. Move the controls before you read the text under each demo, and try the question before the reveal.

The same conditional idea covers sequences. Instead of mapping an input to an output, the model must estimate a distribution over a sequence of variables. A conditional Boltzmann machine can represent the factors of that distribution, each of which predicts the next vector from the ones before it. For a sequence of for times 1 to , the factor at time is a conditional distribution over given all earlier vectors. The conditional RBM of Taylor et al. keeps only the last of them, so its factor has this form:

Click any coloured symbol to see what it means.
Section 2

A conditional RBM over time

A useful sequence model is one that generates realistic motion. Motion capture records the joint angles of actors, and a probabilistic model of those angles allows new animations that were never recorded. Taylor et al. (2007) introduced a conditional RBM that models given the previous values. The model is an RBM over whose bias is a linear function of those past values. The book states this in words. The weights Ak and the offset b0 below are the notation of this page, not the book's.

Click any coloured symbol to see what it means.

The weights never change. Conditioning on different past values gives a new RBM over , and the change in the bias switches different hidden units on or off. Large changes in the distribution over come from that switching, without any change to .

Two visible bits, one past bit, two hidden bits

Pattern of xProbability

Challenge +8 XP

Turn the past bit on and set the scale to 2 or more so that the pattern (1, 0) has probability above 0.8.

Challenge +8 XP

With the past bit on, raise the scale until the pattern (1, 0) has probability above 0.9.

Predict first

The weights of the RBM stay fixed while the past changes. What changes the distribution over the visible vector?

Section 3

Biases or all parameters

In the conditional RBM of Taylor et al., only the bias parameters vary from one time step to the next. The RNN-RBM of Boulanger-Lewandowski et al. (2012) goes further. A recurrent network emits the RBM parameters at each time step, including the weights, so the RBM itself changes shape over time. In the equation below, the symbol θ(t) stands for all of those parameters, and it is this page's notation.

Click any coloured symbol to see what it means.

The counter below compares how many parameters change from step to step in the two designs. The bias-only design changes plus numbers. The full design also changes the weight matrix, which has entries, so it changes that many numbers plus the same biases.

Numbers that change from step to step

Challenge +8 XP

Make the full design change at least 100 numbers per step while the bias-only design changes at most 20.

Challenge +8 XP

Make the full design change more than five times as many numbers as the bias-only design.

Predict first

For 10 visible and 10 hidden units, how many numbers does a bias-only design change per step, and how many does the full design change?

Section 4

Training the RNN-RBM

The loss is not applied to the RNN outputs directly. It is applied to the RBM, so the gradient with respect to the RBM parameters must be estimated with contrastive divergence or a related algorithm. That approximate gradient is then back-propagated through the RNN with back-propagation through time.

Click any coloured symbol to see what it means.
What to take away
  • A conditional Boltzmann machine models structured outputs and sequences through factors such as given the past.
  • The conditional RBM of Taylor et al. keeps its weights fixed and makes its visible bias a linear function of the past values.
  • The RNN-RBM uses a recurrent network to emit all RBM parameters, including the weights.
  • Training the RNN-RBM combines contrastive divergence for the RBM with back-propagation through time for the RNN.
Final quiz

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.7 (pages 685–686), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.