Parameter Tying and Sharing

Tying with a penalty

Picture two teachers who mark the same kind of exam. Their marks should usually agree, but each has a different style. You could force their marking rules to be identical. Or you could add a rule that only pulls them towards each other, and let each keep some of its own style. The second idea is a penalty on the difference between their parameters.

The book considers two models with parameters and , trained on similar tasks with similar input distributions. It suggests that their parameters should be close to each other. The penalty it writes is the squared distance between the two vectors:

The book prints this penalty inline and does not number it. It is shown on its own line here for reference, with no equation number:

The book says that a penalty of this kind is one way to make parameters close, and that the more popular way is to force them to be equal. That second way is parameter sharing, which the next sections study. The book's example is Lasserre et al. (2006), who pulled a supervised classifier towards the parameters of an unsupervised model.

This derivation is added here and is not printed in the book. Take two single parameters and . Each would like to sit at its own target, and , and a penalty with strength pulls them together:

With both derivatives equal to zero, two linear equations remain. Their sum says , so the average does not move. Their difference gives the gap:

The gap shrinks as the penalty grows. It falls towards zero, and the two parameters meet at the average of their targets.

Predict first. As the penalty becomes very large, what happens to the gap between the two tied parameters?

With no penalty, each parameter sits at its own target and the gap is the full difference . Raise the penalty to watch the two curves converge.

Counting the parameters that sharing saves

Parameter sharing is the more popular choice in the book. Each shared parameter is stored once, so only a subset of the parameters needs to sit in memory. The book says this can greatly reduce the memory footprint of a model. It also lets convolutional networks grow without a matching increase in training data.

The count below is added here and is not printed in the book. A dense layer that maps inputs to outputs has its own weight for every pair, plus one bias per output:

A convolution with filters, each of width , stores only one kernel per filter plus one bias. Its count does not depend on the input length at all:

Keep the input long and the convolution stays small. Add filters, and the count grows in proportion to the number of filters, not to the square of the input length.

Shifting the input

The book explains why this works for images. Natural images have many statistical properties that do not change when the picture is translated. A photo of a cat is still a photo of a cat if it moves one pixel to the right. A convolutional network computes the same feature, with the same weights, at every location. So the same cat detector finds the cat at column or at column .

The added formula below writes a one-dimensional convolution on a ring of values. The kernel of width is slid along the input , and the indices wrap around, so the ring has no edges and a shift moves every value by the same amount.

Predict first. If the bump in the input moves three steps to the right, what does the output do?

The readout checks the shift directly. It moves the bump by three places, recomputes the output, and compares it with the original output moved by the same three places. The two agree to rounding.

Sorting the kinds of tying and the summary

The kinds of tying are easy to mix up, so this game sorts them. Each card describes a model. Choose the kind of tying it uses. Four correct answers in a row finish the challenge.

The book's main claim in this section is that sharing is a way to encode what you know about the task into the architecture. A penalty keeps two models close, a constraint makes parts literally identical, and a convolution reuses one kernel everywhere the same feature can appear.

• A penalty on the difference between two sets of parameters ties them together without forcing them equal.
• Parameter sharing forces a set of parameters to be equal, so only one copy needs to be stored.
• For two tied scalar parameters, the gap shrinks as the penalty grows, exactly as the difference of the targets divided by one plus two times the penalty strength (added here).
• A convolution reuses one small kernel at every position, so its count does not grow with the input length.
• A convolution with wrap-around edges is equivariant to translation: shifting the input shifts the output by the same amount.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.9 (pages 253–254), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.