Deep Recurrent Networks

Three blocks in one step

Picture a relay team running laps around a track. The baton is the hidden state , a list of numbers that carries what the team has seen so far. On each lap a runner receives a new input , passes the baton on, and reports an output .

The basic recurrent network in the book has exactly this structure. Equations 10.8 to 10.10 in Section 10.2 describe one step, and their letters are the book’s:

Each step therefore has three blocks, and each block is one weight matrix:

• The input block, with weight , takes the input to the hidden state.
• The state block, with weight , takes the previous hidden state to the new one.
• The output block, with weight , takes the hidden state to the output.

The book calls each block a shallow transformation. The term means one learned affine map followed by a fixed nonlinearity, the kind of transformation that one layer of a deep MLP computes. The question of this section is what happens when each block gets several layers in a row.

Each small circle is a hidden layer and each arrow is an affine map. With zero hidden layers the picture is the basic network of the equations above. Every hidden layer you add puts one extra circle into a block, and every arrow is one more map that the signal must pass. The readout counts the highlighted route.

Depth inside each block

The book asks whether depth would help in each of the three blocks. It reports that experimental evidence strongly suggests so, citing Graves et al. (2013) and Pascanu et al. (2014a). The book adds that this agrees with the idea that a network needs enough depth to perform the required mappings. It also points to earlier work on deep RNNs by Schmidhuber (1992), El Hihi and Bengio (1996) and Jaeger (2007a). This page does not repeat those experiments. The book states the result, and the demos on this page only count routes.

Graves et al. (2013) were the first to show a significant benefit of splitting the hidden state into several layers, as in the book’s Figure 10.13a. In that picture the lower layers turn the raw input into a representation that suits the higher layers. Pascanu et al. (2014a) go further and put a separate MLP, possibly deep, into each of the three blocks, as in Figure 10.13b.

Depth has a cost. The book notes that shallower architectures are easier to optimize, and that a deeper block makes the shortest route between hidden states at different time steps longer. Capacity grows with depth, and ease of training shrinks with it. The bar chart below shows one side of this trade-off with numbers.

A block with hidden layers contains affine maps in series. The first bar counts the maps that carry one input to one output, through the input block and the output block. The second bar counts the maps on the route from one hidden state to the next, which runs through the state block alone. Only the second route crosses from one time step to the next.

Routes and skip connections

Count the edges on the shortest route from to , the hidden state after steps. With hidden layers in the state block, each step costs edges, so the route has the number of edges below.

The book makes the same point for one hidden layer: an MLP with a single hidden layer in the state-to-state transition doubles the shortest route between variables at different time steps, compared with the ordinary network. The general formula above is added here and is not printed in the book. The doubling itself is the book’s.

The book suggests a remedy in Figure 10.13c: put skip connections in the hidden-to-hidden path. A skip connection gives each step a direct edge from the previous hidden state, so the route is back to edges however deep the state block is.

The dashed teal line sits exactly on the no-depth line. That overlap is the point of the skip connection: with it, adding depth to the state block changes nothing about the route.

Which block to deepen

The three blocks do different jobs in the unrolled graph. Deepening the input block adds edges to the route from to , because the input now passes through more maps before it reaches the state. Deepening the output block adds edges to the same route at the other end. Only the state block sits between hidden states, so only it changes the route from to .

Predict before you look. Press the block you think lengthens the route from to . The table then shows each block deepened by one layer, and both routes.

Compare the two columns. The input and output blocks lengthen the route to the output and leave the route between hidden states alone. The state block lengthens the route between hidden states, which is the route that carries memory from one step to the next.

Summary

• The basic recurrent network has three blocks, with weight from input to state, from state to state and from state to output.
• Each block is a shallow transformation: one affine map and a fixed nonlinearity. Hidden layers can be added inside each block.
• The book cites experimental evidence that depth helps, and it names the cost: shallower architectures are easier to optimize.
• A state block with hidden layers makes each step edges long, so one hidden layer doubles the route between time steps.
• A skip connection in the state path restores one edge per step, and depth in the input and output blocks does not touch that route.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.5 (pages 398–399), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.