Inference as Optimization

Why inference is hard

A deep model has visible variables , the data you can see, and latent variables , hidden causes you never observe. Learning needs the posterior, the probability of the hidden causes once the data is known. Some models make the posterior easy to write down. Many do not.

Picture a detective with one clue and two suspects whose alibis depend on each other. Each suspect alone is easy to judge. Once their stories are linked, every combination of answers must be checked together. With many hidden units the number of combinations grows exponentially.

Figure 19.1 of the book shows three graph structures where this happens. Pick one below, then switch on observation to see which links make the posterior hard.

The three panels share one problem in different forms. In the semi-restricted machine, hidden units are linked to each other, so the posterior is one large clique. In the deep Boltzmann machine, neighbouring layers are linked, so the layers cannot be separated. In the directed model, the hidden units have no links at all, but every visible unit has all of them as parents. Once that visible unit is observed, its parents become co-parents, and they explain each other away.

The graph alone does not decide the matter. Probabilistic PCA has the directed structure of the right-hand panel and still has simple inference, because its conditional distributions are linear-Gaussian with mutually orthogonal basis vectors, as the book notes. The graph and the choice of distributions together decide how hard inference is.

The evidence lower bound

The log probability of the data is hard to get when it requires summing over every hidden state. The book replaces the problem with a bound. Pick any distribution over the hidden units. The evidence lower bound, written (ELBO), sits below the true value:

The ELBO is also called the negative variational free energy. The last term is the KL divergence, which measures how far the guess is from the true posterior . It is never negative, and it is zero only when the two distributions match. So the ELBO is never above , and it equals that value exactly when is the true posterior.

Try it on a toy. The data is fixed, and the hidden variable has five states . Each state has a joint value , the probability that the state and the data occur together. The true posterior is , the joint divided by . Set the five weights of with the sliders. The bars show each weight divided by the sum of the five, so they always add to 1. The units are nats, the units of the natural logarithm.

The gap has its own sum, which is the KL divergence of the guess from the posterior. This sum is added here and is not printed in the book.

Two forms of the bound

The bound in equation 19.1 still contains the posterior . The book shows that a few lines of algebra remove it. Press the button to reveal each line and its reason.

The book calls equation 19.7 the more canonical definition of the bound. It needs the joint probability and the entropy of , and never the posterior itself. The check above applies the same algebra to a specific guess, so you can see that the numbers agree.

This derivation follows the book’s equations 19.2 to 19.7 line by line, starting from the definition in equation 19.1. The labels beside each line are added here and are not printed in the book.

Tighter and looser bounds

Exact inference finds the best among all distributions. Approximate inference searches a smaller set of guesses, or stops the search early. Either choice gives a looser bound. A tight bound has a small gap, and a loose one has a large gap.

Imagine the search can only move part way from a plain guess toward the posterior. The weight sets how far it goes. At = 0 the guess is uniform, and at = 1 it is the posterior. Move the slider and watch the gap. The straight path between these two guesses is a teaching device added here. The book does not draw it.

The lower chart shows the gap as a function of . It falls steadily and reaches zero only at the posterior. The book names two sources of looseness. One is a family of guesses too narrow to hold the posterior, which is what the slider limits here. The other is an imperfect optimizer that stops before it finds the best guess, which is like stopping at a smaller .

The bound stays below however loose it is. A poor guess lowers the ELBO but never lifts it above . Inference therefore becomes the job of closing the gap, and learning becomes the job of raising itself.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 19.1 (pages 633–634), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.