Leaky Units and Multiple Time Scales

Many time scales

Picture a river. Ripples on the surface change within seconds. The tide rises and falls over hours. The level of the river drifts over months. A model that reads a sequence faces the same mix. Some details matter for one step only, while a fact from long ago can still matter now. A memory that forgets at a single speed handles one of these badly.

The book names three strategies for giving a recurrent network several speeds: skip connections through time, leaky units with different time constants, and removing some of the short connections. Each one gives part of the model a slower clock. All three answer the problem from Section 10.7, where a gradient carried through many steps shrinks or grows exponentially with the number of steps. This page follows the book’s order.

Predict first. The signal below is a slow tide plus a fast ripple. It is passed through a leaky average with . Which part comes through?

Pick an answer, then move the sliders to test it.

The average follows the signal by the rule . Each new value keeps a share of the old average and adds a share of the new input. This update is written inline in the book and carries no number. The notation is the input at step , and is the running average at the same step.

The share of a wave that survives is its gain . A gain of 1 keeps the wave, and a gain of 0 removes it. For a leaky average with the wave at frequency the gain is

(This formula is added here and is not printed in the book. It follows from the update rule by the usual sinusoid calculation.) At near 0 the gain is near 1 for every frequency, so nothing is removed. As grows, fast waves are cut much more than slow ones, which is why the average behaves like a smoother.

Skip connections through time

Imagine a relay race. A baton passed through every runner is easy to drop, because there are many hand-offs. If it skips every second runner, there are half as many hand-offs. A skip connection does the same for a gradient. It links step directly to an earlier step, with delay steps, so a signal travels over fewer links.

The book credits the idea to Lin et al. (1996). In an ordinary recurrent network a connection goes from step to step . A connection with delay goes from step to step . Each delayed link is one hand-off, and it covers steps at once.

The book states that the gradient now shrinks as a function of rather than , where is the number of steps the signal must cross. The simplified count below shows what this means when each link multiplies the gradient by a weight . The book also warns that single-step connections remain, so the gradient can still explode along them.

Here is the count for a chain of single steps, and for a skip path with delay :

(These two counts are a simplified model added here and are not printed in the book. A path with a last short link needs one more multiplication, so the exponent is rounded up.)

The readout states the two factors at the current path length. Set the delay to 1 and both curves lie on top of each other, because a delay of 1 is an ordinary step. Larger delays lift the skip curve, and the gap grows with the path length.

Leaky units and their time constants

A leaky unit is a unit with a linear self-connection. Its running average is the update from Section 1, and the book uses the coefficient for this self-connection. Near one it remembers for a long time, and near zero it forgets quickly. Here is the same rule again, with the derivative that matters for gradients:

(The book writes this update inline and without a number. The derivative below is added here and is not printed in the book.)

The derivative is at every step. A gradient that travels back through such steps is multiplied by . With near 1, that product stays near 1 for a long time. That is the path the book describes, on which the product of derivatives is close to one.

The half-life tells the same story in units of steps. It is the number of steps after which an old value keeps only half its weight. This formula is also added here and is not printed in the book:

The book also compares leaky units with skip connections. A skip connection needs an integer delay, while is a real number, so the linear self-connection lets a unit adjust how far back it reaches in small smooth steps. The idea goes back to Mozer (1992) and El Hihi and Bengio (1996), and leaky units were also found useful in echo state networks (Jaeger et al., 2007).

The book gives two strategies for setting the time constants. The first fixes them to values that stay constant, for example by sampling them once at initialization. The second makes them free parameters and learns them. The book reports that leaky units at different time scales appear to help with long-term dependencies, and cites Mozer (1992) and Pascanu et al. (2013).

The readout gives the half-life and the gradient factor at the current memory. Try first. The average then equals the input, with no memory, and the half-life is zero. Then move towards 1 and watch the average stretch out.

Removing connections

Skip connections add new links. The book describes a second idea that works in the opposite direction: remove some of the short links, so that the units losing them are forced onto a long time scale. A unit that gains a skip connection may still rely on its short links.

The book gives two concrete routes. One makes the recurrent units leaky, and places groups of them at different fixed time scales (Mozer, 1992; Pascanu et al., 2013). The other updates groups at different frequencies, so that a slow group changes only at some steps (El Hihi and Bengio, 1996; Koutnik et al., 2014). The book reports that the second approach worked well on several benchmark datasets.

The demo below uses the second route. The fast group is a leaky unit with a fixed coefficient of 0.5, and it updates at every step. The slow group updates only when the step number is a multiple of , and holds its value in between. Both groups read the same tide and ripple as Section 1.

Each slow update uses the same leaky rule as before, applied once per interval. Between updates the slow group copies its last value. That is why its line is flat for stretches, and why it follows the tide rather than the ripple. A longer interval means fewer updates, so the slow line changes in fewer, larger steps and reacts more slowly.

• A leaky average with coefficient keeps a share of its old value each step. Its half-life is given by the formula in Section 3.
• A skip connection with delay shortens a path of steps to about links, so the gradient shrinks by a much smaller factor.
• A slow clock comes from removing short connections or from updating groups at intervals, and it costs speed of reaction.
• The time constants of leaky units can be fixed or learned, and the book reports that several scales help with long-term dependencies.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.9 (pages 406–408), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.