The Long Short-Term Memory and Other Gated RNNs

Gates as valves

Picture a bucket with a hole in the bottom. The hole is always open, so the water level falls at a fixed rate whatever arrives. That is a leaky unit, and Section 10.9 used one. Now picture a bucket with three taps. One tap lets water in, one lets the old water out, and one opens the outlet. Each tap is set by the incoming data itself. That is a gated unit.

The book explains why this matters. A leaky unit can accumulate evidence over a long span, but once that evidence is used, the network may need to forget it. Rather than clearing the state by hand, the book wants the network to learn when to do it, and gated RNNs do exactly that.

Predict first. The sequence below is cut into blocks of five steps. A bit of 1 is a hit. The target at each step is the number of hits so far in the current block, and a flag marks the first step of each block. Which fixed or flag-driven strategy follows the target best? This counting task is a toy added here to show the idea. It is not in the book.

Make a guess, then choose a strategy to see how it follows the target.

No fixed leak can restart the count at each block, because the same decay applies at every step. A gate can do it, but only if it can see the block boundary, so this demo gives the gate a flag as an extra input. The learner has to decide which gate setting does the job, and the readout shows how far each one is from the target.

The LSTM cell

The long short-term memory cell has a state unit with a linear self-loop, like a leaky unit. What is new is that the weight on that loop is set by a forget gate . The forget gate is a sigmoid of the current input and the previous output, so its value lies between 0 and 1 and depends on the data. The book also adds an external input gate and an output gate , each built the same way with its own parameters.

Read this as a sigmoid of three things: a bias , a sum of input terms and a sum of recurrent terms . The index runs over the inputs and the hidden outputs. The input gate and the output gate use the same pattern with their own letters.

Figure 10.16 in the book is the block diagram of this cell. The demo below rebuilds it. The input unit and three gates feed two multiplications and one addition, and the state loops back through the forget gate. To keep the picture readable, every input weight is fixed at 4 and the recurrent weights are set to zero, so the self-loop is the only memory path. The book's full equations keep the recurrent terms.

Apart from the input and the previous state, the biases are the only knobs here. Each gate is a sigmoid of four times the input plus its bias. Lower a bias to close a gate, and raise it to open the gate. The readout reports every gate value at the current input.

State update and output

The state is updated by a weighted sum of the old state and a new candidate. The forget gate scales the old state, and the input gate scales the candidate. The candidate is itself a sigmoid of the input in Equation 10.41. The caption of Figure 10.16 notes that the input unit can use any squashing nonlinearity.

The book writes the input unit with the same pattern as the gates, and with its own bias . The output is then a squashed copy of the state, shut off by the output gate:

The output is scaled by . With near 0 the cell stays silent, even though its state is large. The book notes that the state can also feed the three gates, as shown in Figure 10.16, which adds three extra parameters.

Predict first, then move the forget bias.

The forget gate sits at when the input is 0, so each quiet step multiplies the state by about that much. The spike arrives with a larger gate value, and the state then shrinks step by step. A forget bias near 4 keeps most of the spike, and a bias near −4 removes it within a few steps.

Gated recurrent units

The book describes gated recurrent units, or GRUs, as a simpler design. The main change is that one gate does two jobs. The update gate sets the forgetting factor and also the decision to write new content, so there is no separate state. A second gate, the reset gate , decides which parts of the old state feed the new candidate.

The update gate has a copy part, , and a write part, times the new candidate. Both gates are sigmoids:

The book prints the gates of Equations 10.46 and 10.47 with the time index , and uses them one step back, at , in Equation 10.45. The gate that updates the state from step to step therefore reads and . This module follows that reading. (This remark is added here and is not printed in the book. Many papers compute the gates from directly, which is the same unit with the input arriving one step earlier.)

The update gate acts like a leaky integrator whose rate is chosen per unit. At one extreme it copies the old state, and at the other it replaces the state with the candidate. The reset gate changes the candidate by hiding part of the state, which adds a nonlinear link between past and future states.

In the demo every input weight and recurrent weight is 1, the start state is 0.5, and an input spike of 1 arrives at step 1. A high update gate keeps the state almost unchanged, and a low one lets the spike overwrite it.

Other gated RNNs and what matters

The book asks which parts of the LSTM are actually necessary, and what other designs could let a network control the time scale and forgetting of each unit. Its answers come from later work on gated RNNs. The GRU is one such answer. The book reports that several studies of LSTM and GRU variants found no variant that clearly beats both across a wide range of tasks.

Two findings from the book stand out. Greff et al. (2015) found that a crucial ingredient is the forget gate. Jozefowicz et al. (2015) found that adding a bias of 1 to the LSTM forget gate, a practice advocated by Gers et al. (2000), makes the LSTM as strong as the best of the variants they tested. The book also states that, as of its writing, the most effective sequence models in practical applications are gated RNNs.

The demo below does not reproduce either study. It is a single-cell toy that lets you switch each gate off and watch one spike. The spike is followed by a small background signal, so the input gate has something to block.

A switched-off gate is replaced by the value 1, so it lets everything through. A forget gate that is off passes the old state unchanged. An input gate that is off lets every candidate in. An output gate that is off shows the state directly.

• A gated cell uses sigmoid gates between 0 and 1, computed from the input and the past, to decide what to forget, what to write and what to show.
• In the LSTM, the forget gate scales the old state and the input gate scales the new candidate, and the output gate shuts the output off when the state should stay hidden.
• The GRU merges forgetting and writing into one update gate and adds a reset gate for the candidate.
• Switching a gate off changes the memory in a predictable way. The book reports that Greff et al. (2015) found the forget gate a crucial ingredient, and that Jozefowicz et al. (2015) found a forget bias of 1 made the LSTM as strong as the best variants.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.10 (pages 408–412), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.