Section 1
Training criteria
Boltzmann machines are usually trained to approximately maximize the generative criterion . A discriminative RBM instead maximizes , the probability of a label given the visible units. The book notes that this approach often works best with a linear combination of both criteria. RBMs do not seem to be as powerful supervised learners as MLPs, at least with existing methods.
visible units
hidden units
class label
Predict firstWhich quantity does a discriminative RBM train to be large?
How to use this pageClick any colored symbol in an equation or a sentence and it explains itself. Move the controls before you read the text under each demo, and try the question before the reveal.
Click any coloured symbol to see what it means.
The book describes the combination of the two criteria in words. In this page's notation, with a weight between zero and one, the objective to maximize is shown below. The generative part keeps the model a model of the inputs, and the discriminative part pushes it to classify them well.
Click any coloured symbol to see what it means.
Probability of each class for four visible patterns
| Visible pattern | Class 1 | Class 2 |
|---|
Section 2
Higher-order interactions
Most Boltzmann machines used in practice have only second-order interactions. Their energy is a sum of many terms, and each term contains a product of two variables, such as with a weight between one visible and one hidden unit. Higher-order Boltzmann machines let each term contain products of many variables. The energy below adds a binary mask to each visible unit, so the term becomes a product of three variables. This is an illustration in this page's notation, not an equation from the book. Sohn et al. (2013) use masks of this kind, as section 3 describes.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
A mask that removes a visible unit from a hidden unit
| Visible unit | Value | Weight to h | Mask | Contribution |
|---|
Predict firstIn the mask demo, mask 2 is switched off. What does visible unit 2 contribute to the hidden unit?
Section 3
Class interactions and gating
Higher-order terms are used in several ways. Three-way interactions between a hidden unit and two different images can model spatial transformations from one video frame to the next. Multiplication by a one-hot class variable can change the relationship between visible and hidden units depending on the class. Luo et al. (2011) used two groups of hidden units. One group interacts with both the visible units and the class label, and the other interacts only with the visible units. The second group can learn extra features that explain nuisance details, which are needed for realistic samples but do not determine the class.
Predict firstIn the model of Luo et al. (2011), what does the group of hidden units that interacts only with the visible units learn?
Sohn et al. (2013) used a third-order interaction with binary masks on each visible unit. When a mask is zero, it removes the influence of that visible unit on the hidden units. Visible units that are irrelevant to the classification can then be removed from the pathway that estimates the class.
Section 4
An open design space
The Boltzmann machine framework allows many more model structures than have been explored so far. Developing a new form requires more care than developing a new neural network layer, because it is often hard to find an energy function that keeps every needed conditional distribution tractable. Despite that effort, the book notes that the field remains open to innovation.
What to take away
- Boltzmann machines can be trained with the generative criterion or the discriminative criterion , and a linear combination often works best.
- Most practical models use second-order terms, while higher-order terms multiply three or more variables.
- Masks, one-hot class multiplication and separate groups of hidden units let higher-order models gate features or separate class information from nuisance details.
- A new Boltzmann machine needs an energy function that keeps all of its conditionals tractable, which is harder than adding a layer.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.8 (pages 686–687), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.