Multi-Task Learning

Shared and task-specific parts

Think of a kitchen that makes three dishes from one stock. The stock is made once and used by every dish, and each dish then gets its own seasoning. Multi-task learning works the same way. Several related supervised tasks read the same input , share a common middle part of the network, and each finish with their own output layer.

The book describes the common picture in Figure 7.2. The lower layers, reached from the input, form a shared representation . Each task then has its own upper layers, for task 1 and so on, which predict its own target or . The figure also has a factor that is associated with none of the output tasks. The book notes that such factors make sense in the unsupervised setting, where they explain variations in the input that do not matter for or .

The book divides the parameters into two groups:

1. Task-specific parameters. These are the upper layers. They benefit only from the examples of their own task.
2. Generic parameters. These are the shared lower layers. They benefit from the pooled examples of every task.

Switch task 1 off and then on again. The shared layer is the only part that both branches use, so its updates depend on every active task, while each branch answers only to its own task. The factor on the right has no output, so no label ever updates it. This matches the book’s remark that some top-level factors can be associated with none of the output tasks.

Pooling examples for a shared part

The book explains the benefit. Because the shared parameters are used by several tasks, their generalization can improve and their error bounds can improve too (Baxter, 1995). The statistical strength of the shared parts grows with the number of examples that touch them, as long as the sharing is justified.

A short calculation makes this concrete. This calculation is added here and is not printed in the book. Take a slope that is shared by tasks with the same true slope, and measure every task’s inputs around its own mean. Let be the sum of squared spreads of the inputs of task k. With noise of size on each output, a slope estimated from all tasks together has variance

(This formula is added here and is not printed in the book.) With inputs spread evenly over the interval from −1 to 1, the spread is close to a third of the number of examples :

(Also added here and not printed in the book. It is an approximation, and it holds for inputs spread evenly over the interval from −1 to 1, whose variance is 1/3.) If all tasks are equally large, the pooled variance is smaller than the variance of the estimate from task 1 alone by exactly the factor below. Here the error means the variance, because this calculation has no bias.

With equal tasks pooled, the variance is divided by . Four equal tasks cut the variance to a quarter, and eight cut it to an eighth. The next demo lets you check this.

Predict first. With four tasks each of the same size pooled together, what fraction of the single-task variance remains? Press one of the guesses, then move the slider to 4 and read the result.

When the shared part is wrong

Pooling works only when the tasks really share the parameter. The book states the condition: the sharing must be justified, meaning the tasks must have something in common. If task 2 has a slope that differs from task 1, the pooled estimate is pulled towards task 2. The estimate then carries a bias. Its size depends on the share of task 2 among all the examples, so adding examples in the same mix does not shrink it.

(This relation is added here and is not printed in the book.) The pooled estimate is a weighted average of the two slopes, with the weight on task 2 equal to its share of the total spread. The mean squared error of the pooled estimate for task 1 is the squared bias plus the variance.

Task 1 alone has no bias and a variance of . Pooling wins while the squared bias is smaller than the variance it saves. The next demo finds that crossover.

The difference is the gap between the two true slopes, . When it is zero, pooling only adds examples. When it grows, the bias grows with it, and it is larger when task 2 contributes more examples. Use the sliders to find where the pooled curve crosses the flat line of task 1 alone.

Parameter budget and the summary

Sharing also saves memory. Suppose each task has a network with inputs, a hidden layer of width , and one linear output. Training separate networks stores copies of the hidden layer. A shared lower part stores one copy and adds only a small output layer for each task. This count is added here and is not printed in the book.

In the first line each separate network has weights and biases into its hidden layer, then weights and one bias out. In the second line the shared trunk is counted once and each task adds output parameters.

The input width is fixed at 10 in this demo. Each extra task adds a head of parameters to the shared design, but a whole separate network to the other design, so the saving grows with .

• Multi-task learning shares lower layers between related tasks and keeps an output branch for each task.
• Pooling the examples of tasks that share a parameter lowers its variance, roughly by the number of tasks when they are equal.
• The gain holds only while the tasks really share the parameter. A difference between tasks adds a bias that more data in the same mix cannot remove.
• A shared part also saves parameters, because one trunk replaces several copies of the same layers.
• The book’s prior belief is that some of the factors behind the data are shared by two or more tasks.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.7 (pages 244–246), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.