Boltzmann Machines

The big picture

A Boltzmann machine is a model of a probability distribution over binary patterns. Its input is a vector of units, each either off (0) or on (1). The model does not compute outputs from inputs. It assigns a probability to every possible pattern, so it can say which patterns are common and which are rare.

Think of a ball rolling over a landscape of valleys. Low ground is easy to reach and the ball spends most of its time there. A Boltzmann machine works the same way with an energy score: patterns with low energy are likely, and patterns with high energy are rare. The book calls this an energy-based model.

Before the formulas, one prediction. Take three units, so there are 8 possible patterns from 000 to 111. Every energy weight and bias is zero.

Hold on to the answer. The model has no preference yet, so every pattern is equally likely. Each of the next sections adds one ingredient, and the demo in the first of them lets you watch the probabilities move.

Energy and probability

The model assigns each pattern a probability through its energy . The probability is

Here is the partition function, the total of over every pattern. It is the number that makes all the probabilities add up to 1. Lower energy means a larger exponential, and so a larger probability. Adding a constant to every energy changes nothing, because absorbs it.

The energy of a Boltzmann machine is a quadratic function of the units. The book writes it with a weight matrix and a bias vector :

Read the two terms one at a time. The first term rewards pairs of units that are on together when their weight is positive. The second term rewards each unit whose bias is positive when it is on. The book leaves general. This page takes it symmetric with zero diagonal (an assumption added here, not stated in the book). Then the pair term is counted twice in , once for each order of the pair. So the energy contribution of one pair is . The demo below uses the entries as its sliders, so the factor of 2 is included in what you see.

The bars are the probabilities of the eight patterns. Raise and the patterns 110 and 111 both become more likely, because units 1 and 2 are on together in both. Lower and every pattern with unit 3 on, including 111, becomes less likely. The table shows the energy of each pattern, and the bar height is always its exponential divided by the same total.

The demo changes only the energies, and every probability follows from them. The probabilities always add to 1. That is the job of .

Hidden units

With only observed units, the probability of one unit being on is a linear model of the other units, which is logistic regression. So the model can only capture linear relationships among them. Suppose some units are never observed. The book calls them latent or hidden. Split the units into visible units and hidden units . The energy becomes

The matrix links visible units to each other, links hidden units to each other, and links visible units to hidden units. The biases and act on the visible and hidden units. When a hidden unit is on, it can make a pattern of visible units more or less likely together, in a way no single visible unit can. That is why hidden units let the model capture interactions among several visible units. The book states that a Boltzmann machine with hidden units becomes a universal approximator of probability mass functions over discrete variables (Le Roux and Bengio, 2008).

The marginal distribution over the visible units is obtained by adding up the hidden states. For this demo the visible units have no connections among themselves, and the only route to interactions is through the hidden unit. With one hidden unit, weights and a hidden bias , the sum over the two states of gives an unnormalized weight of for each visible pattern. The demo uses that formula, and it is added here and is not printed in the book.

The readout gives the ratio between pattern 101 and pattern 000. The ratio equals at this setting. Notice that the visible units never touch each other, yet the ratio depends on both of their weights together. The hidden unit creates that joint dependence.

Learning by maximum likelihood

Training means choosing and so that the model gives the training data a high probability. The book uses maximum likelihood for this. The learning rule moves each parameter in the direction that raises the log probability of the data. The derivative of the log probability has two parts. The first comes from the data, and the second from the model. The formulas below are added here and are not printed in the book. They come from differentiating equation 20.1 with the energy in equation 20.2. For a pair of units , using the factor of 2 from the energy, they read

and for each bias

Here is the average over the training data, and is the average over the model. The model average is the hard part, because it needs the partition function. Section 5 explains why.

The book makes one observation about this rule that is worth remembering. The update for a weight depends only on the statistics of the two units it connects. Those statistics are collected under two distributions, the data and the model. The rest of the network shapes those statistics, but the weight does not need to know how. The book calls this a local rule. It is also an example of a Hebbian learning rule (Hebb, 1949), often summarized as “fire together, wire together”: a pair of units that are on together more often in the data than in the model has its connection strengthened. The book notes that this makes the learning rule somewhat plausible as a model of the brain, and that the negative, model-driven phase is harder to explain biologically. It also says that dream sleep may be a form of negative phase sampling, and it labels this idea speculative.

The table is the local rule in action. Each row concerns one pair of units. The change in a weight depends only on that row, the data column minus the model column. Reset the model and read the table at the start. The change column is the step that the next press would make. Its largest entry belongs to the pair whose co-activation the data shows most strongly beyond what the model expects.

The partition function obstacle

Both the probability in equation 20.1 and the model average in the learning rule need the partition function . The sum in runs over every binary pattern, so its size doubles with each unit added. The sum of over all states has to be recomputed whenever the parameters change. The book says so directly: all Boltzmann machines have an intractable partition function, so the maximum likelihood gradient must be approximated, using the methods of chapter 18.

The demo below counts the patterns. Each extra unit doubles the work for an exact sum. The three-unit demo above is cheap, but a model with 30 units has more than a billion patterns, and each one needs its own term.

The next section, on restricted Boltzmann machines, removes the connections within each layer. The book says that this gives conditional distributions that are factorial and relatively simple to compute and to sample from.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.1 (pages 654–656), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.