Transfer Learning and Domain Adaptation

Sharing the right layers

Transfer learning starts from a simple question. If you learned something in one setting, can it help in another? Often it can, when the two settings share some underlying factors. Cats and dogs, ants and wasps: both sets of pictures contain edges, shapes and changes of light. A network that learned to see edges on the first set has a head start on the second.

The sharing can happen in two places, and the right place depends on what stays the same. The book calls the two cases transfer learning and domain adaptation, and they both use one shared representation . Pick a scenario and look at which blocks are shared.

Predict first. A speech system must output valid sentences, but the sound of the same word differs between speakers. In that case, which part should be shared?

Choose one answer.

The rule is simple. If the inputs differ from task to task, each task needs its own lower layers to turn its input into generic features. The upper layers are then shared, and they learn what the shared outputs need. If the inputs mean the same thing but the outputs differ, the lower layers can be shared instead. This structure is what the book draws in its Figure 15.2 with a selection switch. The switch chooses the task-specific block that reads each input.

The core idea of representation learning is that the same representation may be useful in both settings. Using it in both lets the representation benefit from the training data of both tasks.

Domain adaptation and a shift in the inputs

In domain adaptation the task stays the same, but the input distribution changes. The book’s example is sentiment analysis. A predictor trained on reviews of books and films is used on reviews of phones. The idea of a positive or negative comment is the same, but the vocabulary and style differ. Unsupervised pretraining with denoising autoencoders helped in this setting.

A simple model, added here, shows the effect. Call the first setting and the second . In the positive class has centre and the negative class has centre , with = +1 and = −1. Both classes have spread = 0.5. In both centres move by a shift , while the spread stays the same. A fixed boundary predicts the class of each input. The accuracy on the new setting is

This closed form for a two-class Gaussian toy is added here and is not printed in the book. Here Φ is the cumulative distribution of the standard normal: Φ(z) is the chance that a standard normal value is below z. The two terms are the chance that a positive input lands above the boundary and the chance that a negative input lands below it. Move the shift and the boundary, and watch the accuracy.

Predict first. The shift is 1, so both centres now sit at 0 and 2. Which boundary keeps the accuracy high?

Choose one answer.

The book also names concept drift: a slow change of the distribution over time, which is a form of transfer learning. Both concept drift and transfer learning can be seen as special cases of multi-task learning. The phrase multi-task learning usually refers to supervised tasks, while transfer learning also applies to unsupervised and reinforcement learning. Their common goal is to use data from the first setting when predicting in the second.

One-shot learning: one example per class

Deeper representations also reduce the labelled data that the transfer task needs. In two 2011 transfer learning competitions, the deeper the representation learned from the first setting, the better the learning curve on the new categories, so fewer labelled examples were needed for the same generalization (Mesnil et al., 2011; Goodfellow et al., 2011).

Two extreme cases remain. In one-shot learning the transfer task gives one labelled example. In zero-shot learning, which is the next section and which the book also calls zero-data learning, it gives none. One-shot learning works when the first stage has already cleaned up the representation. If the classes are cleanly separated, then a single example per class is enough to label every test example that lands near it.

Place one labelled example for each class, and each point takes the label of its nearest example. The dots are the test examples. The background shows which example would claim each spot.

Drag the large dots. The accuracy depends on where each example sits, not on how many points there are. The test points cluster around their class, so one well-placed example is enough. The book says the same: one labelled example suffices because the representation has already gathered each class into a cluster around one point. This works to the extent that the factors that matter have been separated from the others.

Predict next. How many labelled examples does one-shot learning use per class?

Choose one answer.

Zero-shot learning through a shared space

Imagine a learner that has read a large amount of text saying that cats have four legs and pointy ears. It has never seen a picture of a cat, yet it can guess that a new image shows one. Zero-shot learning makes this possible when the text and the images share a representation, and the text describes the object well enough.

The book describes the setting with a task variable . The model estimates the conditional distribution

so the question has an input , an answer and a description of what to ask. The description must be a distributed code, not a one-hot label, so that similar tasks have similar codes. Words provide this: the word for each category has a learned vector.

The shared-space recipe has three parts, as in the book’s Figure 15.3. Each domain has its own encoder, the images have and the words have . A map links the two spaces, and it is learned from pairs of matched examples. Here is the recipe in a toy. The six classes have image representations in one plane and word vectors in another. The last class is never shown with its label. The map is fitted from the classes that were seen, and then the unseen images are mapped and matched to the nearest word vector. The book does not print this formula. The equation below is added here to make the toy exact.

The book’s point is that the three ingredients, the two representations and the link between them, should ideally be learned together. In the toy the two representations are fixed, and only the link is fitted. That is a deliberate simplification. The result still shows how more labelled pairs make the link more accurate, and how a link fitted on too few pairs fails completely.

Zero-shot learning is a particular form of transfer learning. The book adds that the same principle explains multi-modal learning, where a representation in one modality is anchored by the other.

Translation and paired modalities

Machine translation is the same idea in language. Words of one language and words of another can each be given a distributed representation from unlabelled text. A few translated sentence pairs then link the two spaces. Even when the pair “word A to word B” never appears in the labelled data, the link can guess the translation, because both words sit near their neighbours in their own space.

The book notes that this transfer works best when all three ingredients are learned jointly: the two representations and the relation between them. Learning them separately, as the toy above does for the map, is therefore a simplification.

Multi-modal learning follows the same pattern. A model learns a representation for images, one for text and a relation for matched pairs, so a concept in one modality is anchored in the other. The book uses pairs of observations from two modalities for this, and the result is a model that can generalize to new pairs.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 15.2 (pages 536–541), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.