The Deep Learning Approach to Structured Probabilistic Models

Same tools, different choices

Deep learning uses the same basic tools as other work with structured probabilistic models: graphs, factors, energy functions and sampling. The designs differ. Think of a traditional graphical model as a hand-wired circuit, where a designer chooses every wire. A deep model is closer to a line of identical stages, where each pair of stages is joined by one table of weights.

The chapter uses two kinds of variables. A latent variable is one the data never records directly. An observed variable is one the data does record. Each statement below describes either a traditional graphical model or the deep learning approach. Sort them before you read the reveal.

The last row is the most unusual choice. A traditional model simplifies until every quantity can be computed exactly, because the exact computation keeps inference (answering questions about a model's variables) tractable, meaning it can be finished in reasonable time. The deep approach keeps the full model and accepts approximate answers. The book says that such models often have marginal distributions that cannot be computed, and that the practitioner is satisfied to draw approximate samples from them. A marginal distribution is the distribution of a few variables after the others are added out.

The book puts the deep approach this way: find the minimum amount of information that is absolutely needed, then find a reasonable approximation of that information as quickly as possible. The rest of this page follows one small example of that approach: the restricted Boltzmann machine (RBM) from section 16.7.1 of the book. The toy in this page has 3 visible units and 2 hidden units. It is added here, and the book does not print its numbers.

Depth counted in the graph

A graph shows which variables interact directly. Each variable is a point, and each direct interaction is a line between two points. The depth of a latent variable is measured in that picture. It is the number of links on the shortest path from the latent variable to an observed variable. A link is one step. The depth of the model is the greatest depth of any of its latent variables.

Picture a relay in which each runner hands the baton to the next. A latent variable that hands over directly to an observed variable is at depth 1. One that needs two handovers is at depth 2. Write the rule in symbols as a restatement, which the book does not print:

The demo starts with three links on. Here is at depth 1, and is at depth 2 through . The unit has no path to an observed unit, so it does not count yet. The depth counts links, not units. A path with three links has depth 3, even though it passes through four units.

The book separates this depth from the depth of the computation. A deep network can run many layers of computation inside one conditional distribution, while its graph has only one layer of latent variables. The book notes that many generative models used in deep learning have no latent variables, or only one layer of them, but use a deep computation to define each conditional distribution.

Deep models usually have more latent variables than observed variables. Their interactions flow through several latent variables, so the depth of the graph can grow with the number of layers. Traditional models have few latent variables, and each one often has a meaning chosen in advance.

Layers joined by matrices

A distributed representation describes an input by many units switched on together, rather than one unit for one concept. Deep models connect each visible unit to very many hidden units, so the hidden layer gives a distributed representation of the visible units. One input is spread over many units at once.

This density has a price for classical graph methods. The book says the graphs become too dense for exact inference, and for loopy belief propagation, an approximate method that works well on sparse graphs, meaning graphs with few links. Deep learning almost never uses loopy belief propagation. Most deep models are instead designed so that Gibbs sampling, which draws each variable from its conditional distribution given the others, or variational inference, which fits a simpler distribution to a hard one, is efficient.

Units are therefore grouped into layers, and the links between two layers are described by one matrix. The number of weights for one pair of layers is the number of visible units times the number of hidden units:

The grid below uses = 3 visible units and = 4 hidden units. It is a separate example for this point, not the toy of section 4. Each tick is one link, which is one weight in the matrix. A matrix product runs fast on hardware built for it. The book notes that sparsely connected generalizations, such as block diagonal products or convolutions, keep these steps efficient.

A dense product costs times multiplications whatever the zeros are. A sparse format stores and multiplies only the links that are on. The number of weights that must be multiplied falls with the links, which is why sparse structure is worth having. A sparse store keeps memory for the links that are on, and nothing else.

The energy of a restricted Boltzmann machine

The RBM has binary visible units and binary hidden units . Each unit is 0 or 1. It is an energy-based model: it gives every full state an energy , and a lower energy means a higher probability. The book's energy is:

The book states that , and are real-valued parameters that learning adjusts. The first term is a bias, a number that shifts the input of each visible unit. The second term is a bias on each hidden unit, and the last term couples the two layers through the matrix . Each term carries a minus sign, so a positive weight lowers the energy when the two units it joins are both on.

The probability of a full state follows from the energy. The number is the partition function, the sum of the exponential of minus the energy over all states. It makes the probabilities add up to 1.

The toy has 3 visible and 2 hidden units, so it has joint states. That is small enough to add all 32 terms in the demo. The real models in the book cannot do this, which is the reason for the sampling in section 6.

The word restricted means there are no links between two visible units and none between two hidden units. The book's Figure 16.14 draws the RBM as a Markov network, a graph with undirected links, here with two groups of nodes and links only between the groups. That figure is not reproduced on this page. The picture below redraws its structure for the toy. Click one unit, then another, to test whether a link is allowed.

Conditionals that factorize

The restriction gives two properties. Given all the visible units, the hidden units are independent of one another. Given all the hidden units, the visible units are independent of one another. The book writes them as (16.11) and (16.12):

A distribution factorizes when it is a product of simpler factors. Read (16.11) as a product. The conditional distribution of the whole hidden layer is a product of one factor per hidden unit, and each factor is the probability of one hidden unit given the visible layer. Equation (16.12) has the same form with the roles swapped. Each factor is a single number between 0 and 1, so the whole layer is cheap to describe.

For binary units, each factor is a sigmoid. The logistic sigmoid, written , turns any real number into a value between 0 and 1. It is 0.5 at zero, close to 1 for large positive inputs, and close to 0 for large negative inputs. The book gives the probability that hidden unit is on and that it is off:

The book prints in both equations. The energy (16.10) names the hidden bias . This page treats the printed as a misprint and writes for the hidden bias in the same place. The input to the sigmoid is the pre-activation of the unit: the weighted sum of the visible units plus its bias. The visible-side conditional follows from the same argument with the roles swapped. This line is added here and is not printed in the book:

The table shows the exact conditional of the whole layer, computed from the joint terms, next to the product of sigmoids from (16.11) or (16.12). The two columns agree to rounding error. That agreement is the factorization. The sigmoid plot shows where each unit sits on the curve. A unit near the middle of the curve is uncertain, and one at either end is almost sure.

Block Gibbs sampling

Gibbs sampling draws one variable at a time from its conditional distribution. The factorization of section 5 lets the chain draw a whole layer at once, which is block Gibbs sampling. One sweep draws all the hidden units given the visible units, and then all the visible units given the hidden units. The book uses such chains to draw approximate samples when the exact sums are out of reach.

A chain's states follow the model only after it has run long enough. The first states still remember the starting point, so a short run can be biased. The demo below keeps this idea on the toy. The exact distribution over the 8 visible states is known there, because the 32 joint terms can be added. Figure 16.15 of the book shows samples from an RBM trained on MNIST, a set of handwritten digits, drawn by Gibbs sampling. That figure is not reproduced on this page. Successive samples in the book's figure are highly correlated. The demo uses the toy because the trained model cannot be reproduced here.

To measure how far the sampled distribution is from the exact one, this page uses the total variation distance. It is an added measure and is not printed in the book. It is 0 when two distributions are the same and 1 when they share no mass:

The chain starts at 000 and runs for the number of sweeps shown. The readout and the trace lines give the visible state after each sweep. The bars show how often the chain visits each state. A short run can miss a state that carries much of the exact probability, and a long run follows the exact bars more closely, although one run can rise again for some lengths.

Derivatives and learned features

Training needs derivatives. The energy is linear in each weight, so its derivative is simple. The book gives the derivative of the energy with respect to one weight as (16.15):

The derivative equals minus the product of the two units that the weight joins. It is when both units are on, and 0 when either one is off. The book states that undirected models like this one can be trained by computing such derivatives on samples from the model. That is why the sampling of section 6 matters to training.

The demo changes the weight by a tiny amount in each direction, measures how the energy moves, and compares the slope with the formula. The two numbers agree because the energy is linear in the weight. A derivative of 0 means the weight has no effect on the energy in that state.

After training, the representation of a visible state is the expected hidden vector, the average of the hidden units under their distribution given that state. The book uses the expectation over the hidden layer as a set of features for the input. Each entry is the probability that a hidden unit is on, so the features are the sigmoids of section 5:

The demo shows these features for the chosen visible state. Each feature is a number between 0 and 1. The book says that training induces a representation h of the data v, and that the expected hidden vector can serve as a set of features that describe v.

• The deep approach keeps the full model and accepts approximations, such as approximate samples and approximate gradients.
• The depth of a latent variable counts links on its shortest path to an observed variable, which is not the depth of the computation.
• One matrix per pair of layers gives fast products, but the dense graphs they form leave exact inference and loopy belief propagation unhelpful.
• The RBM energy has no links inside a layer, so its conditionals factorize into sigmoids, equations (16.11) to (16.14).
• Block Gibbs sampling alternates the two layers. Early states remember the start, and longer chains move closer to the exact distribution.
• The derivative (16.15) is minus the product of the two units a weight joins, so it is read directly from the states of those units.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 16.7 (pages 585–589), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.