Restricted Boltzmann Machines

Two layers with no links inside

A restricted Boltzmann machine (RBM) is a Boltzmann machine with one layer of visible units and one layer of hidden units. The word restricted means that there are no links inside either layer. Each visible unit connects to hidden units, and each hidden unit connects to visible units, and nothing else. The idea goes back to Smolensky (1986), who called it a harmonium. RBMs are common building blocks because the restriction makes them much easier to work with than a general Boltzmann machine.

Picture two rows of people at a meeting. Each person in the top row talks only to people in the bottom row, and never to a neighbour in their own row. Once the bottom row has settled on a pattern, each person in the top row makes an independent choice, so they can still end up in different states, but none of them is swayed by the others. That last point is the key to everything this page builds.

The answer is the key property of the RBM, and the rest of the page shows why it holds. The first part of the reasoning is the structure, which the next section draws.

The bipartite graph and the energy

The graph of an RBM is bipartite: its nodes split into two groups, and every edge joins a node of one group to a node of the other. Nothing joins two visible units or two hidden units. Figure 20.1a of the book draws this graph with four hidden units above three visible units. The demo below rebuilds that picture with sliders for the number of units.

The number of links between the layers is . In the book's figure every visible unit links to every hidden unit, so the graph is complete between the two layers and empty inside each layer. The book notes that sparsely connected RBMs are also possible, such as convolutional RBMs.

Its energy is the Boltzmann machine energy with the visible-visible and hidden-hidden weights set to zero. The book writes the joint distribution as

with the energy function

Read the three terms of the energy. The term and the term are biases, one per unit. The last term is the interaction, and it is a sum over every visible-hidden pair. Each entry links one visible unit to one hidden unit. A large positive entry makes the two units prefer to be on together, and a large negative entry makes them prefer opposite states.

The normalizing constant is the sum over both layers:

The book states that Long and Servedio (2010) proved this partition function intractable for RBMs. The sum has terms, so cannot be computed exactly for realistic sizes. The probability of a visible pattern alone, obtained by summing out the hidden units, shares that problem. The next section shows what the bipartite structure does buy.

Conditional distributions

Even though is intractable, the conditional distribution of the hidden units given the visible units is simple. The derivation below follows the book. Start from the definition of a conditional probability.

The denominator does not depend on , so it only rescales the numerator. Substitute the joint distribution from equation 20.4 into the numerator. The result is equation 20.8:

The factor does not depend on , so it is absorbed into the normalizer. Equation 20.9 keeps only the terms that matter for the hidden units, with the constants written as a new normalizer , as the book does.

The hidden units appear in a sum of single-unit terms, so the exponential factors into one product per hidden unit. This factorization is the whole point. Each factor involves only one and the weights of that unit. Equation 20.10 is the sum form and equation 20.11 the product form.

Exponentiating a sum gives a product, so equation 20.11 writes the same expression with one factor per hidden unit:

Each factor is an unnormalized distribution over one binary unit, so normalizing it is simple. The book writes the normalized probability that unit is on as

Substituting the two exponentials gives equation 20.13, a ratio with the unit off in the denominator:

which is a logistic sigmoid. The sigmoid turns any real number into a probability between 0 and 1.

Because the factor for each hidden unit depends only on that unit, the full conditional is a product of sigmoids. The book writes it compactly, with a sign trick. Multiplying by turns a 0 into minus one and a 1 into plus one, so one formula covers both states:

For the argument is , and for the argument is its negative. Since , the formula gives the correct probability for both states. The same argument works for the visible units given the hidden units, and the book states the result as equation 20.16.

Turn the visible units on and off and watch the table. Each row is a hidden pattern, and the two columns always agree, up to rounding. Each hidden unit's probability depends on all the visible units, through its own column of . Changing one visible unit shifts the sigmoid input of every hidden unit, and it does not touch the other hidden units directly, because no link joins two hidden units.

Training with sampling

Training an RBM by maximum likelihood needs the gradient of the log probability, and that gradient has two parts. The data part is easy, because it averages over training examples. The model part needs samples from , and the partition function is intractable, so that part cannot be computed exactly. The book says that an RBM can be trained with any of the techniques of chapter 18 for models with intractable partition functions. These include contrastive divergence (CD), stochastic maximum likelihood (SML, also called persistent contrastive divergence or PCD), and ratio matching.

The book singles out one property. Compared with other undirected models, the RBM is relatively simple to train, because the conditionals and can be computed exactly in closed form and are simple to sample from. The book notes that a deep Boltzmann machine combines an intractable partition function with intractable inference, so it lacks this property.

Sampling alternates the two blocks. This is block Gibbs sampling. Draw every hidden unit from its conditional, then draw every visible unit from its conditional, and repeat. Each pass of the two steps is one sweep. After many sweeps, the visible states the chain visits are drawn from the model distribution, up to the burn-in time that the chain needs to forget where it started.

The bars show two things. The black bars are the exact probabilities, found by adding up every hidden state for each visible pattern. The violet bars are how often the chain visited each pattern since the last reset. With few sweeps the violet bars are rough, and they settle as the sweeps accumulate. Changing any weight or bias resets the chain, because the exact distribution has changed.

The last challenge asks for a setting in which one pattern dominates. The exact bars respond at once to a slider, but the visited bars start again from zero after every change, so the chain must run again before its bars match the new exact ones.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.2 (pages 656–660), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.