Greedy Layer-Wise Unsupervised Pretraining

A representation that makes the next task easy

Picture two rings of stones on a table. The dark stones form a small ring in the middle, and the light stones form a larger ring around it. No straight ruler can put the dark stones on one side and the light stones on the other. A pair of compasses can, because the two rings differ in how far they are from the middle.

That is the core idea of a representation: a new description of the same data, chosen so that the next task becomes easy. The book’s chapter opening says the same thing in one line: a good representation is one that makes a later learning task easier. A supervised deep network already does this. Its last layer is a simple linear classifier, and the layers below it learn a representation that the classifier can use. Here is the raw input, and the produced from it is a list of numbers that carries the information the classifier needs.

Predict first. You want a rule that puts the dark stones on one side and the light stones on the other. Which rule works?

Choose one answer.

Now test it. Each stone is a point, and its representation is one number, its squared distance from the middle. The left panel is the table. The right panel is the representation, one number per point, with a dashed threshold . A point is called light when its is above the threshold.

Each point has . This example is added here and is not printed in the book. The dark ring has between about 0.36 and 1.0, and the light ring has between about 2.56 and 4.0, so any threshold between those values separates the rings perfectly. No straight line on the table can do the same job, which is why the representation carries the work.

Greedy, one layer at a time

Greedy layer-wise unsupervised pretraining takes a deep network and trains it one layer at a time, without labels. It is greedy because each piece is optimized on its own, and layer-wise because the pieces are the layers. The book states Algorithm 15.1 in general terms. Here it is in its core form.

A single-layer learner takes a training set and returns an encoder . The first layer is trained on the raw data . Then its output becomes the data for the next layer:

The composed encoder grows by one stage each time:

Here is the data at the current stage, counts the stages from 1 to , and the circle means that the new layer is applied after the old ones. Once a layer is trained, the book’s procedure keeps it fixed while the next layer is trained. The lower layers are trained first and are not adapted again when the upper layers arrive. The protocol can end with an optional fine-tuning step that trains the whole stack together with the labels, but the pretraining itself uses no labels at all.

The word greedy describes how the procedure optimizes: each piece is optimized on its own, one piece at a time, and the pieces already trained are not revisited. The cost is that a lower layer chosen early cannot adapt to the layers added later. The book calls the whole procedure pretraining because it is meant to be only a first step before joint training of all the layers.

The same greedy procedure can also initialize deep autoencoders and deep generative models with many layers of latent variables, such as deep belief networks and deep Boltzmann machines. Chapter 20 describes those models. The book also mentions greedy supervised pretraining, which builds on the idea that a shallow network is easier to train than a deep one (section 8.7.4).

Why pretraining helps: priors and features

The book gives two ideas for why pretraining helps. The first is that the starting parameters of a deep network can regularize it (and, to a lesser extent, improve optimization). The second is that learning how the inputs are distributed can help with the mapping from inputs to outputs. The second idea is easier to see, so start there.

The book is clear that pretraining does not always help. Ma et al. (2015) found that, on average, it was slightly harmful for predicting chemical activity, although it was significantly helpful for many tasks. Before using it, the book asks when and why it works.

Take the example in the book: a generative model of car and motorcycle images must know about wheels and how many wheels appear in a picture. A feature for wheels is useful for the unsupervised task and may also be useful for the classification task. Whether that happens depends on the task, and the book admits that this is not yet understood mathematically.

Words are the clearest case. Represent each word by a one-hot vector , a vector of zeros with a single one in the position of that word. Every pair of different words then has the same distance:

So the squared distance between any two different words is 2. One-hot vectors say nothing about which words are alike. A learned embedding can place similar words close together. Drag the words below and watch the distances change, then compare with the one-hot matrix on the left.

Pretraining therefore has more to offer when the starting representation is poor. The book says pretraining is less useful for images than for words, perhaps because images already lie in a rich vector space where distances give a low-quality similarity measure.

Predict next. Pretraining is a regularizer as well as a source of features. When does the book say it helps most?

Choose one answer.

Where the walk ends: the starting point

Figure 15.1 in the book shows many networks, each trained from a different random start. The figure places each network, at one moment of its training, as a point in function space. Networks without pretraining end in one region, and pretrained networks end in another, smaller region. The book suggests that pretraining reduces the variance of the training result. Erhan et al. (2010) also found that pretraining reduced the mean and variance of the test error most for deeper networks.

The picture below is a toy, not a real network. It has one parameter , a loss with two valleys, and a walk that starts where you put it. The walk follows the rule of gradient descent:

Here is the learning rate and is the loss. The loss has a deeper valley on the left and a shallower one on the right, with a hump between them. Where the walk starts decides which valley it falls into. Local minima are no longer considered a serious problem for real networks, as the book says, so this toy illustrates only the role of the starting point.

Predict first. You start at 0.5, which lies right of the hump. Where does the walk go with the default learning rate?

Choose one answer.

The book’s point carries over to pretraining. An initialization is a choice of starting place, and a starting place that lies on a different side of the hump changes where the walk ends. Pretraining is one such choice, made by an unsupervised rule rather than by chance.

What it costs: two phases, many knobs

Pretraining and fine-tuning are two training phases, and each one has its own settings. Joint training of an unsupervised and a supervised cost in one stage has a single weight on the unsupervised cost. The book lists two disadvantages of the two-phase design, and the sorting task below shows both.

Sort the knobs. For each item, tap it once for one combined knob (joint training) and again for one of many, in a phase (two-phase pretraining). Tap a third time to clear it.

The book’s arguments are these. With simultaneous training, one coefficient sets how strongly the unsupervised objective regularizes the model, and a smaller coefficient always means less regularization. With pretraining, the strength is not something you can adjust smoothly. The starting parameters are either the pretrained ones or not. The second phase cannot be predicted during the first, so the most principled way to choose the pretraining settings is to use the validation error of the supervised phase. In practice, some settings, such as the number of pretraining iterations, are chosen by early stopping on the unsupervised objective, which is cheaper but not ideal.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 15.1 (pages 528–536), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.