Semi-Supervised Disentangling of Causal Factors

Causes that the representation separates

Earlier sections asked what makes one representation better than another. This section gives one answer: an ideal representation separates the causes that generated the data, so that each direction in the representation stands for one cause. The book writes the causes as and the observed data as . When a label is one of those causes, predicting it is easy from a representation that has separated the causes.

Many learning methods start from a different goal, a representation that is easy to model, for example sparse or with independent entries. The book’s hope is that for many AI tasks these two goals coincide: once the causes are found, they are easy to read off.

Here is a toy version of separating causes. Two independent causes produce the data by mixing, so each observed coordinate depends on both causes at once. A representation that disentangles the causes returns each cause on its own coordinate. Turn the representation until each coordinate follows one cause. The representation is a rotation of the causes by an angle :

The rotation is the only thing you control, and it is the book’s whole idea in miniature. At an angle of 0 the first coordinate is exactly the first cause, and the score is 1. At 45 degrees each coordinate blends both causes, and the score falls to about 0.71. The score is the average, over the two coordinates, of the largest correlation with any cause. Its value for a rotation of θ is max(cos θ, sin θ), which is cos θ up to 45 degrees. The rotation and the score are added here and are not printed in the book.

When p(x) reveals the classes

Why would unlabelled data help at all? The book gives a clear case. Suppose the true process has a hidden cause , and the data is generated from it: first a cause, then the observation. Written as a directed model with as the parent of :

The observed data then has the marginal distribution

The best model of is therefore the one that finds the hidden . If the label is one of these causes, Bayes’ rule ties it to the marginal:

The point is in the denominator. The conditional is built from the same pieces as the marginal , so learning the shape of from unlabelled data tells you something about the classes. This works when the class is closely tied to one of the main causes. Figure 15.4 in the book shows this case, a mixture with one component per class.

The book raises a complication. Most observations have a very large number of underlying causes, and the learner does not know which one is the label. The brute force remedy is to capture and disentangle every salient cause, which is not feasible in practice. Two strategies are used instead: a supervised signal alongside the unsupervised one, so that the model keeps the causes that matter for the labels, or much larger representations.

Predict first. Three well-separated bumps of data come from three classes, and you have no labels at all. What does modelling the shape of the data alone tell you?

Choose one answer.

Each bump has spread = 1 and weight = 1/3 in this toy, and the bumps are centred at −d, 0 and d. The readout gives the posterior of Equation 15.3 for each bump at the marked input, and the accuracy comes from giving every input to the bump with the largest posterior. The toy has no labels in the fit. The bumps are known here only because we built the toy. The readout reports them as the classes.

Salience: what the reconstruction error notices

The book raises a practical problem. Autoencoders and generative models have historically been trained with a fixed criterion, often similar to mean squared error on the pixels. That criterion decides which causes are salient, meaning important enough to encode. A cause that changes many pixels a lot is salient. A cause that changes few pixels a little is not, even if a robot must interact with it. Humans show a similar selectivity: they often miss changes that are not relevant to the task they are doing (Simons and Levin, 1998).

Figure 15.5 in the book shows an autoencoder trained with squared error on a robotics task. It fails to reconstruct a small ping pong ball, while the robot can interact with larger objects such as baseballs, which squared error finds more salient. The toy below shows the same arithmetic. It is a made-up image, not a trained network, and it shows what the squared error counts.

Here is the average over all pixels, and is the reconstruction of pixel . Forgetting an object means the reconstruction shows background instead, so each lost pixel costs its brightness change squared. A big object with a small change, and a small object with a big change, can cost the same.

Predict first. Which quantity does the squared error add up?

Choose one answer.

The book gives one concrete way to change what counts as salient: a learned cost. Generative adversarial networks train a generator to fool a classifier that tells generated samples from training samples, so any structured pattern the classifier can recognize becomes salient. In Figure 15.6 a model trained with squared error alone drops the ears of a head, because they hardly change the brightness. The same model trained with the adversarial term keeps them, because they follow a predictable pattern. Section 20.10.4 of the book describes this framework in detail.

Causal direction and robustness

The book gives one more reason to model the causes. If the true process runs from cause to effect , then the conditional describes a mechanism that stays put. The laws of the world do not change when the mix of causes changes. Only the marginal weights move, so changes while does not.

Bayes’ rule shows the consequence. Equation 15.3 says the posterior depends on the prior. If the prior changes, the posterior changes with it, although the mechanism underneath does not. A model of the mechanism can therefore be corrected for a new prior directly, through Bayes’ rule.

The book adds a caveat. If the arrow runs the other way, so that causes , then is sensitive to changes in , and the robustness argument fails. The direction of the causal arrow decides which piece is stable.

Move the prior below. The two class curves stay fixed, and the mixture moves. Watch the posterior at the marked input.

The book states the consequence in one sentence: the causal mechanism remains invariant while the marginal distribution over the causes can change, so a model of the causes generalizes better and is more robust. The demo lets you see both parts of that sentence at once.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 15.3 (pages 541–546), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.