Learning about Dependencies

Hidden variables carry dependencies

Two shops on the same street often have busy days together. Neither shop tells the other what to expect. The weather does: on a sunny day both shops fill up, and on a rainy day both stay quiet. In this story the weather is a hidden variable, because nobody records it in the shop books. Whether each shop is busy on a given day is a visible variable.

A generative model has to capture how the visible variables depend on each other. The standard tool in deep learning is to add hidden variables . Two visible variables and then become linked through a hidden variable, with a direct link from to and another from to :

In this picture the two shops never talk to each other, yet their busy days move together. The next demo uses one hidden bit , which is 1 on a sunny day and 0 on a cloudy one. The chance of a sunny day is . Each shop’s chance of a busy day depends only on the weather.

The readout gives the correlation between the two shops, a number from −1 to 1 that measures how far they move together. In the all days view it is positive, because both shops respond to the sun in the same direction. Switch to one kind of day with the tabs. On a single kind of day the correlation falls to zero. Once the weather is fixed, the shops are independent in this model. The link between them ran through the weather alone.

The book states the general rule. A hidden variable can make two visible variables dependent, while each of them is independent of the other once the hidden value is known. For the two shops the covariance follows from the same bit. This closed form is derived for this page and is not printed in the book:

The covariance divided by the spread of each shop gives the correlation . The formula shows what the demo showed. The dependence needs a hidden bit that moves both shops, and it disappears when the chance of a sunny day is 0 or 1, because then the weather never varies.

The cost of connecting everything

A direct link between every pair of visible variables looks harmless. It is not. The book says that a model without hidden variables needs many parents per node in a Bayesian network, or very large cliques in a Markov network. A clique is a group of variables that all interact directly with one another, and the table that describes it grows exponentially with its size.

Take a clique of binary variables, each 0 or 1. A table with one number for every pattern of those variables has entries. The entries must add up to one, so one of them is fixed by the others. The number of free numbers, which this page derives and the book does not print, is:

Now give one hidden bit a link to each of the visible bits. Once is known, the visible bits are independent of each other. The model needs one number for the chance of the hidden bit, and two numbers for each visible bit, one for each value of . This count is derived for this page and is not printed in the book:

Each extra visible bit doubles the full table and adds only two numbers to the hidden variable model. The vertical axis is a logarithm, so a straight line means a multiplication by a fixed factor. The table line keeps climbing, while the hidden variable line flattens out on the scale.

The cost has two parts. The first is computational: every entry must be stored, and a sum over the table visits every entry. The second is statistical: a model with an exponential number of parameters needs a matching amount of data to estimate them well. A hidden variable model with a small count needs less data, but it can express only the dependencies it has room for. The book’s remedy is to connect only the variables that are tightly coupled, and to omit the edges between the rest.

The cost of searching for structure

The choice of which edges to keep is a problem of its own, called structure learning. The book says that most structure learning methods are a form of greedy search, which takes one best small step at a time. The book describes the loop in prose, and this page numbers its four steps:

1. Propose a structure, which is a set of edges.
2. Train a model with that structure.
3. Give the trained model a score. The score rewards high accuracy on the training set and penalizes model complexity.
4. Propose small changes, each adding or removing a few edges, and repeat with the best one.

Every candidate needs its own training run, and the search may take many rounds. The demo counts the candidates in one round. Each cell of the triangle is one pair of variables. Click a cell to switch that edge on or off, and watch the count.

Each pair of variables can be toggled exactly once in a round. The number of single-edge changes is therefore , the number of pairs, and it does not depend on which edges are already present. This count is derived for this page and is not printed in the book:

The count grows with the square of , so every round trains about models. With rounds, the search trains models in total, on the assumption that each round tries every single-edge change. A real search may try fewer. The book sets this cost against latent variables, which avoid both the discrete search and the repeated rounds of training.

A fixed structure with hidden variables

The book’s alternative keeps one fixed structure. Each visible variable links directly to a hidden variable, and the visible variables are indirectly linked through it. The book says that ordinary parameter learning then produces the right dependencies in the marginal distribution . The demo shows what one hidden bit can express.

Give each visible variable a link strength between −1 and 1. A positive strength makes the visible bit more likely to be 1 when = 1. A negative strength reverses that. A strength of 0 cuts the link, which the graph draws as a dashed line. The chance of 1 is when = 0 and when = 1, built as 0.5 minus or plus 0.4 times the strength. This mapping is a choice made for this page, not a formula from the book.

The covariance of two visible variables follows from the same bit. The formula is derived for this page and is not printed in the book:

The sign of every correlation is therefore the product of the two link signs. A link of zero switches off every correlation through that variable. With equal to 0.5, each correlation is 0.64 times the product of the two strengths. One hidden bit can produce only this product pattern, so the demo is a limited picture of what the book means. Real data may need many hidden variables to match its dependencies.

Latent variables as features

A hidden variable can also be read off. It gives a second description of the input. The book notes that in a mixture of Gaussians, the latent variable records which category an example came from, and so can serve for classification. The book also reports that , the expected hidden value given the visible value, or the most probable hidden value, is often a good feature for .

Take two clusters with centres and , a shared spread and a chance of cluster 1. Here means the input came from cluster 1. The feature for one input is the posterior chance of cluster 1, the chance once the input is known. The density of the input under each cluster, a bell-shaped Gaussian curve, is written with and :

The centres sit at and , so the distance between them is . This is the standard Bayes rule for two Gaussian clusters, written out for this page. It is not a printed formula in the book.

The feature is one number between 0 and 1 for each input. A threshold at 0.5 turns it into a classifier. In this toy the clusters are known in advance. In a deep model the latent variables are learned from the data, and the feature is learned with them.

• A hidden variable links two visible variables indirectly. Once the hidden value is known, the visible ones can become independent.
• A full table over binary variables needs numbers. A hidden variable with one link to each visible variable needs .
• A search over structures trains about candidate models per round, and the count grows with the square of the number of variables.
• With one fixed hidden variable, the sign of each correlation is the product of the two link signs. Richer patterns need more hidden variables.
• The posterior of a hidden variable is a feature. It is a number between 0 and 1 that can feed a classifier.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 16.5 (pages 582–584), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.