Section 1
Undirected layers with no links inside
A deep Boltzmann machine (DBM) is an entirely undirected model with several layers of latent variables. Unlike the RBM, it has more than one hidden layer. Like the RBM, every unit in a layer is independent of the others in that layer once the neighbouring layers are fixed. Figure 20.2 of the book draws a DBM with one visible layer and two hidden layers. Its joint distribution is an energy-based form, and the book leaves out the biases for now.
visible units
, hidden layers
, weights
energy
Predict firstIn a DBM with three hidden layers, which links are present?
How to use this pageClick any colored symbol in an equation or a sentence and it explains itself. Move the controls before you read the text under each demo, and try the question before the reveal.
Figure 20.2, interactive
Challenge +8 XPSet the picture to show exactly 24 links in total, counting both pairs of neighbouring layers.
hidden unitvisible unitlink
The joint distribution is defined by an energy function, with a normalizer that sums over every layer:
Click any coloured symbol to see what it means.
For a DBM with one visible layer and three hidden layers, the energy is
Click any coloured symbol to see what it means.
Compared with equation 20.5 of the RBM, the DBM energy has extra weight matrices between the hidden layers. As later sections show, these connections change both the behaviour of the model and how inference works.
Section 2
Bipartite structure and Gibbs sampling
The layers of a DBM form a bipartite graph when the odd layers sit on one side and the even layers on the other. Figure 20.3 of the book rearranges the graph this way. Condition on the even layers, and the odd layers become conditionally independent. Condition on the odd layers, and the even layers do the same. The RBM conditionals therefore carry over to every layer, and the book gives them for the two-hidden-layer case:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Gibbs sampling then needs only two blocks. One block holds all the even layers, including the visible layer, and the other holds all the odd layers. Each block is sampled in one step, so the chain updates the whole model in two iterations rather than one per layer. Efficient sampling matters most for stochastic maximum likelihood training.
The bipartite view of the same DBM
Challenge +8 XPSet hidden layer 2 to exactly 3 units, with the other two layers at their default sizes. The picture then has 24 links in total.
Predict firstHow many blocks does Gibbs sampling need in a DBM with three hidden layers?
Section 3
Mean-field inference
The posterior over the hidden layers of a DBM does not factorize, because the weights between the hidden layers couple them. Its shape is still simple enough to approximate with a mean-field family, in which the hidden units are independent. The approximation is a Bernoulli product with one parameter per unit, and the goal is to make it as close as possible to the true posterior. The closeness is measured by the Kullback-Leibler divergence:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The parameters of the approximation are the mean-field estimates and , each the probability that a unit is on. The book writes the approximation as a product of Bernoulli terms:
Click any coloured symbol to see what it means.
At a fixed point of the update equations, the approximation is a local maximum of the variational lower bound. The two updates alternate, first the first layer and then the second:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
On small problems such as MNIST, as few as ten iterations can be enough to find an approximate positive phase gradient, and fifty usually give a high quality representation of one example. The demo below runs these updates and compares them with the exact marginals, which you can compute here because the model is small.
Mean-field updates against the exact marginals
Predict firstIf the weights between the two hidden layers are all zero, is the mean-field estimate exact?
CheckMean field gives an approximation of the posterior, so the gap is not a bug. In a DBM, the approximation still includes the top-down feedback from deeper layers, which the upward pass of a DBN ignores.
Section 4
Learning, pretraining and joint training
Learning in a DBM faces two problems at once. The partition function is intractable, and so is the posterior. The variational lower bound on the log likelihood handles the second problem:
Click any coloured symbol to see what it means.
The partition function still appears, so evaluating the probability requires approximate methods such as annealed importance sampling. DBMs are typically trained with stochastic maximum likelihood. Contrastive divergence is slow for a DBM, because hidden units cannot be sampled efficiently given the visible units. The book gives the variational stochastic maximum likelihood procedure as Algorithm 20.1. Its main steps are:
Algorithm 20.1, summarized
- Sample a minibatch and arrange it as the rows of a matrix . Start the hidden estimates, possibly at the model marginals.
- Run the mean-field loop until it converges: set the first layer to of its input from and the second layer, then set the second layer to of its input from the first.
- Set the positive phase gradients from the data and the mean-field estimates.
- Run Gibbs steps, in two blocks, starting from persistent samples of the model, to form the negative phase.
- Subtract the negative phase gradients, and update the weights by a step of size .
Training a DBM from random weights with stochastic maximum likelihood often fails. The model may fail to represent the distribution, or it may represent it no better than an RBM. The original remedy is greedy layer-wise pretraining. Each layer is trained as an RBM, the first on the data and each later one on samples from the posterior of the previous one. The stack is then combined into a DBM. In a stack of three or more RBMs, the middle weights are halved, and the DBM is trained with persistent contrastive divergence. Figure 20.4 of the book shows this procedure as four stages.
Figure 20.4, the pretraining stages
Challenge +8 XPReach the stage where the RBMs are combined into a DBM, and set the factor for the middle weights to 0.5, the value the book uses.
Joint training avoids pretraining. The centered DBM subtracts a fixed vector from all states. This reparametrization does not change which distributions the model can represent, but it often gives a better conditioned Hessian, so the model can train without a greedy stage. The uncentered energy is equation 20.36:
Click any coloured symbol to see what it means.
The centered energy is equation 20.37:
Click any coloured symbol to see what it means.
Equation 20.36 is the special case = 0 of equation 20.37. The multi-prediction DBM is the other route.
It treats the mean-field equations as recurrent networks that predict missing values, and it trains them by back-propagation. Each example keeps a random subset of its variables as inputs, and the remaining variables become targets.
Figure 20.5, a schematic of inputs and targets
This is a schematic, not a copy of figure 20.5. The book's figure has one column per unrolled mean-field step, while this grid has one column per variable. Each row is one example in a minibatch. Click a cell to switch it between input (black) and target (gray). Inputs stay fixed, and the targets are what the recurrent network must predict.
Challenge +8 XPMake every row have exactly two inputs and exactly two targets.
What to take away
- A DBM has undirected links between every pair of neighbouring layers, and its layers form a bipartite graph.
- Gibbs sampling updates all even layers in one block and all odd layers in the other.
- Mean-field fixed points approximate the posterior. The gap comes from interactions between hidden layers.
- Training uses variational stochastic maximum likelihood after layer-wise pretraining, or a joint method such as centering or multi-prediction.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.4 (pages 663–676), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.