Boltzmann Machines for Real-Valued Data

Gaussian-Bernoulli RBMs

Binary pixels are a poor fit for natural images, so the book turns to models of real-valued data. The most common choice is an RBM with binary hidden units and real-valued visible units, where the visible units are Gaussian with a mean that depends on the hidden units. The book uses a precision formulation, in which the visible conditional is

Only the terms that involve belong in the energy. Expanding the log of the Gaussian gives

The term collects everything that depends only on the parameters, and the normalizer absorbs it. The expansion also contains a term that couples pairs of hidden units, . Including it would turn the model into a linear factor model, so the book omits the pairs and keeps the energy an RBM. Each single hidden unit still gets a term, equation 20.41, which is optional and changes the learning dynamics.

With the precision matrix diagonal, the energy of the model is

The demo uses one real-valued visible unit, two binary hidden units and a bias for each hidden unit. The dashed curve is the conditional density for the state you chose. The solid purple curve is the marginal density, which adds up every hidden state.

Conditional covariance with mcRBM and mPoT

The Gaussian RBM only models the conditional mean of the input. Ranzato and colleagues argue that natural images carry much of their information in the covariance between pixels, so a model with a fixed covariance misses it. The mean and covariance RBM (mcRBM) splits its hidden layer into mean units and covariance units. The mean units form a Gaussian RBM, and the covariance units form a covariance RBM. The energy is the sum of the two parts:

The joint distribution and the conditional over the observations follow. The conditional is a multivariate Gaussian whose covariance is not diagonal, so the covariance units control how the pixels co-vary:

Sampling the observations given the hidden units would need the inverse of at every step, which is expensive. Training uses hybrid Monte Carlo on the marginal instead. The mean-product of Student's t-distributions (mPoT) model follows the same pattern, with Gamma distributions over the covariance units in place of binary units:

Each active covariance unit adds a term along its direction to the precision. The variance along that direction then shrinks, which is what the energy in equation 20.45 asks for. The demo draws the resulting ellipse.

The spike and slab RBM

The spike and slab RBM (ssRBM) also models covariance, with a different mechanism. It has two sets of hidden units. The binary spike units decide whether a component appears, and the real-valued slab units set its intensity. The mean of the visible units is , and when a spike is active, its slab adds variance along the matching column. Its energy has the form

where is the offset of spike , is a diagonal precision on the observations, is a scalar precision for slab , is a nonnegative diagonal penalty, and is the slab mean. Marginalizing out the slabs gives a Gaussian over the observations:

The matrix is the inverse of a sum that subtracts for each active spike. That is why turning a spike on widens the distribution along its component. The book notes that is a valid covariance only when this matrix is positive definite, and that some parameter settings break this condition.

The demo sets the observation precision to the identity and the penalty to zero, so only the spike terms change the covariance. The ellipse is the covariance when the precision is positive definite. When it is not, the book's warning applies, and the density has no finite normalizer.

The cost of covariance

The three covariance models differ in what they need at each step. The mcRBM and mPoT conditionals need the inverse of a dense covariance, and their training depends on hybrid Monte Carlo. The ssRBM needs no matrix inversion, and contrastive divergence and persistent contrastive divergence with Gibbs sampling still apply. The ssRBM pays with the positive definiteness condition, which the book says has no fully satisfying solution yet. Convolutional ssRBMs produce good natural image samples.

The comparison is an order of magnitude estimate, not a timing. Inverting a dense matrix takes work on the order of . Applying a diagonal precision takes work on the order of . Their ratio is , which is why dense covariance models are hard to sample at image sizes.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.5 (pages 676–683), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.