Section 1
Stacked layers with two kinds of link
Deep belief networks (DBNs) are generative models with several layers of latent variables. The latent variables are usually binary, and the visible units may be binary or real. There are no links inside a layer. The top two layers are joined by undirected links, and every other pair of neighbouring layers is joined by directed links that point toward the layer closest to the data. Figure 20.1b of the book draws this shape. Usually every unit in a layer connects to every unit in the neighbouring layers, although sparser DBNs are possible. The book notes that DBNs, introduced in 2006, were among the first non-convolutional deep models to train successfully, and that they have since fallen out of favour.
visible units and biases
hidden layers
weight matrices
sigmoid
Predict firstIn a DBN with two hidden layers, which links are undirected?
How to use this pageClick any colored symbol in an equation or a sentence and it explains itself. Move the controls before you read the text under each demo, and try the question before the reveal.
Figure 20.1b, interactive
Challenge +8 XPSet the picture so the top two layers have exactly 12 undirected links between them.
hidden unitvisible unitdirected link
The DBN with hidden layers contains weight matrices, to , and bias vectors, one for each layer including the visible layer. The book writes the joint distribution of the top pair as equation 20.17, and that pair is an undirected RBM. Every other pair of neighbouring layers is directed, so the lower layers are described by conditionals, not by a joint, as section 2 shows. Equation 20.17 reads
Click any coloured symbol to see what it means.
Read the exponent as three parts. The term and the term are biases, and the last term couples the two layers through the weights of layer . The coupling is the only interaction in the top pair.
Section 2
Conditionals and sampling
Each directed link gives a conditional that is a product of independent sigmoids. A unit in a middle layer depends on the layer above it, and a visible unit depends on the first hidden layer. The book writes the conditional of each unit in layer , and the conditional of each visible unit, as
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
For real-valued visible units, the sigmoid is replaced by a Gaussian with mean and diagonal precision , which is easy to work with:
Click any coloured symbol to see what it means.
To generate a sample from a DBN, first run several steps of Gibbs sampling on the top two hidden layers. That stage draws a sample from the RBM formed by those layers. Then one pass of ancestral sampling through the rest of the model draws the visible units. A DBN with only one hidden layer is just an RBM, so this procedure reduces to the RBM sampler of the previous section when there is one hidden layer.
Sampling a small DBN against its exact distribution
Challenge +8 XPDraw at least 2000 samples and make every sampled frequency lie within 0.03 of its exact probability.
exact probabilitysampled frequency
The demo uses a visible layer of two units, a first hidden layer of two units and a second hidden layer of two units. All biases are zero to keep the page short, and the equations keep them. Each sample runs one Gibbs sweep on the top pair and then draws the visible units from equation 20.19. The bars come from adding up every state of the hidden layers, so the exact values are known. The sampled frequencies settle toward them as the samples accumulate.
Why inference is hardThe book lists two sources of difficulty. Inference is intractable because of explaining away within each directed layer: when several hidden units could each explain what the visible units show, learning that one of them is on makes the others less likely, so the units cannot be treated separately. Inference is also hard because the two undirected top layers interact. The evidence lower bound is intractable too, and so is the log likelihood, which needs the partition function of the top pair. The book notes that the log likelihood can still be approximated with annealed importance sampling (Salakhutdinov and Murray, 2008).
Predict firstTo draw a sample from a DBN, which step comes first?
Section 3
Greedy layer-wise training
A DBN is trained one layer at a time. First, train an RBM on the data to maximize the expected log probability of the visible units, using contrastive divergence or stochastic maximum likelihood. The parameters of this RBM become the parameters of the first layer of the DBN. Next, train a second RBM so that it models the hidden units of the first RBM, when the first RBM is driven by the data. The book states this as approximately maximizing
Click any coloured symbol to see what it means.
where is the distribution of the first RBM and is the distribution of the second. The second RBM sees samples of the first hidden layer, and it learns to model them. The same step can repeat to add as many layers as desired, and the procedure can be justified as increasing a variational lower bound on the log likelihood of the data.
In most applications, no effort is made to train the whole DBN jointly after the greedy procedure. The book notes that generative fine-tuning with the wake-sleep algorithm is possible.
What the second RBM has to model
| First hidden layer | Under the data |
|---|
The data has four visible patterns, with counts 00: 4, 01: 2, 10: 1 and 11: 3. The first RBM has the weights below and no biases of its own on the visible layer.
Challenge +8 XPSet the first RBM so that the data puts at least 0.5 on the first hidden layer being 1 1.
Predict firstAfter the first RBM is trained, what does the second RBM learn to model?
Section 4
Fine-tuning as an MLP
The trained DBN can be used directly as a generative model, but most of the interest in DBNs came from their use in classification. The book takes the weights from the DBN and uses them to define a multilayer perceptron. The first layer computes
Click any coloured symbol to see what it means.
and each later layer computes
Click any coloured symbol to see what it means.
After the MLP is initialized with the generative weights and biases, it is trained on a classification task. This extra training is an example of discriminative fine-tuning. The choice of MLP is a heuristic that works well in practice. It is not derived from a tight bound. The MLP propagates information upward from the visible units, and it does not send information downward or sideways, while the DBN itself has explaining-away effects within each layer and top-down effects between layers.
One upward pass through the MLP
Predict firstIn the upward pass of equation 20.22, can the second hidden layer change the first hidden layer?
CheckThe term deep belief network is often used for any deep network. Strictly, it means a model with undirected links in the deepest pair of layers and directed links pointing down between all other pairs of neighbouring layers. The name also overlaps with dynamic Bayesian networks, which share the acronym DBN.
What to take away
- A DBN stacks RBMs. The top two layers are undirected, and every lower link is directed toward the data.
- Sampling runs block Gibbs sampling on the top pair, then one pass of ancestral sampling down to the visible units.
- Training is greedy and layer by layer. Each new RBM models the hidden units of the layer below, driven by the data.
- The weights define an MLP for classification, and the upward pass ignores the downward and sideways interactions of the DBN.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.3 (pages 660–662), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.