Section 0
The big picture
The recurrent networks of the last two modules read the input from left to right. The state at step holds information about the past and the present, and nothing about the future. The book calls this a causal structure.
Many tasks need more than that. The book gives speech recognition as an example: the correct reading of the current sound may depend on the next few sounds, because of co-articulation, and on the next few words. Handwriting recognition has the same property. A bidirectional recurrent network is built for this case. It has two passes over the sequence, one forward and one backward, and the output at each step can use both.
Predict first. The learner only sees the sequence one symbol at a time, but the target at step depends on the next symbol. On random bits, what accuracy does a model that copies the current symbol get on average?
Choose an accuracy to see the answer.
Predict the next bit from the past, and from both sides
The target at step t is the bit at step t plus 1, so the target depends on the future. The two models are both toy rules chosen for this page and are not trained. Both read observed bits, and noise flips some of them: a random draw flips each observed bit with probability 0.25, and a flipped bit has a dashed border. The causal model copies the observed bit it read at the previous step. The bidirectional model reads the observed bit at step t plus 1 through its backward state.
True bits x(1) to x(8)
Observed bits, dashed where noise flipped them
Causal copy model, prediction for steps 1 to 7
Bidirectional model, prediction for steps 1 to 7
Challenge +8 XPPress Alternating bits. The causal copy model must be right at none of the seven steps.
A causal state can never give the bidirectional advantage, because the information about is not in the state at step . The backward pass is the book’s way to bring that information back.
Section 1
Two passes over the sequence
The book describes the bidirectional network in Figure 10.11. A forward recurrence computes from left to right. This is the same update as equation 10.5, written for one step at a time:
Click any coloured symbol to see what it means.
A second recurrence, which the book calls the recurrence, runs from right to left and starts at the end of the sequence. Its form is shown below. The book describes this pass in words and figure only, so the display is a sketch and is not printed in the book:
Click any coloured symbol to see what it means.
The output then uses both states. The book says that the output units can compute a representation that depends on both the past and the future. The book does not print a combining equation, so the equation below shows one form used in this module. This derivation is added here and is not printed in the book.
Click any coloured symbol to see what it means.
Predict first. In a bidirectional network with five inputs, which inputs can affect the output at step 3?
Choose an answer to see the result.
Inputs that reach one output
forward state hbackward state goutput o
The demo draws the two chains. The forward chain covers the inputs up to step , and the backward chain covers the inputs from step to the end. Each output therefore has a path to every input, and the paths differ in length.
Section 2
How much each input counts
The book says the output is most sensitive to the inputs around time , without a fixed window. A toy version makes that statement exact. Take a scalar state with a memory for the forward pass and for the backward pass, and let the output be the sum of the two states minus the input. This toy is added here and is not printed in the book:
Click any coloured symbol to see what it means.
Under these rules the output at step is sensitive to input with a size that follows the display below. This derivation is added here and is not printed in the book.
Click any coloured symbol to see what it means.
Predict first. With both memories equal to 0.5 and the output at step 4, which input has the largest sensitivity?
Choose an answer to see the result.
Sensitivity of one output to each input
Challenge +8 XPSet the output step to 3 and choose the memories so that input 1 counts for more than input 5 by more than 0.1.
Challenge +8 XPSet the output step to 4 and choose the backward memory so that input 7 counts for more than 0.3.
With , the bars are symmetric around the output step. The peak stays at the output step, because that input always has sensitivity 1. Making one memory longer than the other makes the bars on that side fall more slowly, so the output leans toward that side. The toy shows why the book’s wording is about the inputs near : the sensitivity decays on both sides, at a rate set by the two memories.
Section 3
Beyond sequences: grids
The book extends the idea to two-dimensional input, such as images. Four recurrent networks scan the grid, one in each direction: up, down, left and right. Each output at a grid position can then combine information from all four scans. The book notes that this is more expensive than a convolutional network, but it allows long-range interactions between features in the same feature map.
Predict first. How many scan directions does the book use for a two-dimensional input?
Choose a number to see the answer.
Cells that reach one output through the four scans
This picture is added here and is not printed in the book. It shows the cells that lie on a straight line from the chosen cell in each direction. The book describes the four scans without this count.
Challenge +8 XPSet the output cell to row 3 and column 3, with all four scans on. The count of reached cells must be 9.
In principle the four scans let the output see the whole row and column through the cell. The book says the output could also depend on long-range inputs, if the RNN is able to learn to carry that information along the scan.
- A causal state holds the past and the present, and cannot use the future.
- A bidirectional network runs a forward and a backward pass, so each output can use the inputs on both sides.
- The sensitivity of an output to an input falls with the distance between them, at a rate set by the memories of the two passes.
- For two-dimensional input, four scans cover the four directions, and the book notes that this costs more than a convolution.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.3 (pages 394–395), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.