Providing Clues to Discover Underlying Causes

Labels and priors as clues

Chapter 15 asks what makes one representation better than another. The book answers that a good representation disentangles the underlying causal factors behind the data, which means each factor is separated from the others. Learning needs clues that point at those factors.

A supervised label is the strongest clue. It comes with each input, and it often names one factor directly. Unlabelled data gives weaker hints. These hints are priors: implicit beliefs about which explanations are likely, imposed by the designer of the learning algorithm.

Predict first. There are four yes or no factors, A, B, C and D, which makes 16 candidate explanations. A witness says that factor A is on. How many candidates remain?

Choose one answer.

A label is hard evidence. Every candidate that disagrees with it is removed, and the rest keep their weights. This equation is added here; the book states the idea in words only:

Set each factor to unknown, on or off in the demo. The grid lists all 16 candidates. Grey cells disagree with a label. A prior does not remove anything. It sets the weights of the candidates that remain, and the demo uses a simple prior that makes each factor on with a fixed probability.

The same three labels can pick a different winner when the prior changes. Set A on, B off and C off, then switch between the prior that favors few factors on and the one that favors many. The winner moves from one candidate to its neighbor that differs only in D. The book says it is impossible to find a universally superior regularization strategy, so the choice of prior is a real decision.

Smoothness and linearity

The first generic strategy is smoothness: nearby inputs give nearby outputs. The book writes it as follows, where is a unit direction and is a small number:

Smoothness lets a learner carry what it knows from training examples to nearby points. The book adds that it is insufficient to overcome the curse of dimensionality on its own.

The second is linearity. Many algorithms assume that relationships between variables are linear. The demo below uses the predictor , where is the slope and the intercept. This simplest linear form is added here; the book does not print it. The book says linear predictors can predict far from the observed data, but may give extreme values there.

The two assumptions are different. Linear functions with large weights in a high dimensional space may not be very smooth. The demo puts both on the same training data.

Predict first. The training points lie between −1 and 1. The marker will move to x = 3, which is outside the data. Which predictor lands closer to the true value there?

Choose one answer.

The local average follows the data closely inside the training range and stays near the last training values outside it. It does not run away, but it cannot follow a trend either. The straight line follows the trend and runs away from the truth. Neither is right everywhere, so the book treats each as an assumption about the world, not as a fact.

Causes, depth and hierarchy

The book lists several more clues. In the first, the data comes from multiple explanatory factors, and most tasks become easy once the state of each factor is known. Learning then needs features that also help , because both refer to the same factors.

The causal clue asks the model to treat the factors as the causes of , and not the other way round. The book says this helps semi-supervised learning and makes the model more robust when the distribution over the causes changes, or when the model is used for a new task.

Predict first. The chance that the hidden cause is 1 rises from 0.2 to 0.8. Which part of the picture changes?

Choose one answer.

The toy below has one hidden cause that is 0 or 1. Given the cause, is drawn from a bump centered at 0 or at 2. The mixing weight is the chance that the cause is 1. This model is added here to make the causal clue concrete, and it is not printed in the book:

The bumps stay where they are because the mechanism that turns a cause into an observation does not change. Only the weights move. A causal model keeps the mechanism and absorbs changes in the cause distribution, which is why the book finds it more robust.

The last clue in this group is depth, read as a hierarchy. High-level abstract concepts are defined in terms of simpler concepts, and the book describes a deep architecture as a belief that the task is solved by a multi-step program, where each step refers back to the output of the steps before it. The demo builds such a hierarchy one level at a time.

Shared factors across tasks

Suppose several tasks share one input. Each output may depend on its own subset of a common pool of factors . The subsets overlap. Learning every through one shared representation lets the tasks share statistical strength, as the book puts it.

Call the factors used by task A and those used by task B . The shared pool needs one factor for each member of the union, and a factor that both tasks use is counted once. Counting this way gives the identity, which is added here and is not printed in the book:

Predict first. Task A uses factors 1, 2 and 3. Task B uses factors 3, 4 and 5. How many factors does a shared pool need?

Choose one answer.

The shared pool saves exactly the overlap. Separate models would learn each task's factors on their own, and the shared pool learns each factor once. The book describes the benefit as statistical strength flowing between tasks, which is the reason to look for overlap before building separate models.

Manifolds, clusters and coherence

The manifold clue says that probability mass concentrates in regions that are locally connected and occupy a tiny volume. Those regions can be approximated by low dimensional manifolds inside the space where the data lives. A circle drawn on a flat sheet is a one dimensional curve in a two dimensional plane. Points close to it fill only a thin band.

For a band of half width around the unit circle, inside a square of side 3, the share of the square that the band covers follows from the area of the annulus:

This formula is added here, and it is not printed in the book. Move the band width below and read the share.

The clustering clue says that each connected manifold can be given one class, and the class stays the same along it. The data may lie on many disconnected manifolds, so a decision boundary belongs in the empty space between them. The demo uses two parallel lines of 20 points. Each point's nearest neighbor is found by distance in the plane.

The spacing along each line is 1/19, about 0.053. When the gap is wider than that spacing, a point's nearest neighbor is on its own line, and the rule works. When the gap is narrower, a neighbor across the gap is closer than the next point on the same line. The fix is to use the structure of the manifold, not the raw distance.

The coherence clue says that the most important factors change slowly, or at least that they are easier to predict than raw observations such as pixel values. The demo builds an observation from a slow factor plus fast noise. Averaging over neighboring times recovers the factor, and the readout measures by how much.

Sparsity and simple dependencies

The sparsity clue says that most features should be absent for most inputs. A feature that can be present or absent should be absent most of the time. A cat image does not need a detector for elephant trunks. A simple way to write this as a prior is a probability that each of the eight features is on, with the features independent of each other:

This form is added here to make the sparsity prior concrete. It is not printed in the book. Each code gets the product of one factor per feature, so a code with few features on has a large probability when is small.

Predict first. At p = 0.1, which code is more probable: one feature on, or four features on?

Choose one answer.

The simplest dependence between factors is marginal independence. The book writes it as a product over the factors, and it notes that linear dependencies, or those captured by a shallow autoencoder, are also reasonable assumptions, and that such dependencies can be seen in many laws of physics.

For two factors, the largest gap between the joint probability and the product of the single probabilities measures how far the pair is from independent. This measure is added here and is not printed in the book. The demo sets the correlation of two binary factors that each have probability one half:

Representation learning ties together the many forms of deep learning, and the book closes the chapter by calling learning the best possible representation an exciting avenue of research.

• A label removes the candidates that disagree with it, and a prior sets the weights of the candidates that remain.
• Smoothness lets a learner carry knowledge to nearby points, and linearity lets it predict far away, which can be extreme.
• Causal factors make a model robust when the distribution over causes changes, because the mechanism stays put.
• Shared factors save the overlap between tasks, manifolds and clusters explain where a boundary belongs, and coherence favors slow factors.
• Sparsity favors codes with few features on, and marginal independence reduces a joint distribution to a product of single factors.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 15.6 (pages 554–557), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.