Section 0
The big picture
Translation, question answering and speech recognition share one property: the input and the output sequences in the training set are generally not the same length, although their lengths may be related. The earlier wirings of this chapter tie the two lengths together. The book writes the constraint as for the wirings of Figures 10.3, 10.4, 10.10 and 10.11.
The encoder-decoder architecture removes that constraint. An encoder RNN reads the input sequence. A decoder RNN writes the output sequence. The two are joined by a context, a single summary of the input, shown in Figure 10.12.
Predict first. The input has five words and the output must have seven. Can a plain wiring like Figure 10.3 produce it?
Choose an answer to see the result.
Two lengths, two wirings
Challenge +8 XPSwitch to the plain wiring and make the output longer than the input. The step count can no longer match.
The demo uses the same two sliders for both wirings. The plain wiring has one output step for each input step, so any difference in length leaves steps without a partner. The encoder-decoder has no such pairing, because the decoder starts from the context and runs for however many steps it needs.
Section 1
The encoder and the context
The encoder is an RNN that reads for steps 1 to . The book says that the last state of the encoder is typically used as the context:
Click any coloured symbol to see what it means.
The book describes this as printed text, without an equation number, and it is shown here as a display for reference. The context is then given to the decoder, so the decoder sees one summary of the whole input.
Predict first. When the input grows from 3 steps to 8 steps, what happens to the size of the context?
Choose an answer to see the result.
Read an input and keep the last state
The encoder here is a toy: each state is the hyperbolic tangent of a memory times the previous state plus the input, and the inputs are plus or minus 1. The toy and its memory value are chosen for this page. This derivation is added here and is not printed in the book.
Input steps
Challenge +8 XPBuild an input of six steps whose context is above 0.5. Use the plus and minus buttons.
The toy confirms the book’s wording about the context. However many input steps are read, the context is one value here, and in general one fixed-size vector. The memory controls how far back the context remembers, which is the same idea as the memory of Module 10.3.
Section 2
The decoder
The decoder is an RNN that generates for steps 1 to . The book says it is conditioned on the context in the same way as the vector-to-sequence RNN of Figure 10.9, so the context can enter at the initial state, at every step, or both. The two RNNs are trained jointly. The book describes the training goal in the text, as the average log probability of the output sequence given the input sequence, over the training pairs:
Click any coloured symbol to see what it means.
The lengths and are free. The decoder runs for steps whatever the encoder read.
Predict first. If the decoder is asked for 3 output words from a context, how many decoder steps run?
Choose an answer to see the result.
Run the decoder from the context
The decoder here is a toy: its state is the hyperbolic tangent of 0.8 times its previous state plus the context, starting from zero. The 0.8 is chosen for this page. This derivation is added here and is not printed in the book.
Challenge +8 XPGive the encoder two input steps in the encoder demo, then set the decoder to eight output steps.
The decoder does not need the same number of steps as the encoder. The length of the output is a choice made when the decoder runs, and the context has no length of its own.
Section 3
A fixed-size context and attention
The book names a clear limitation. When the context has too few numbers to summarize a long input, the decoder loses information. The book cites Bahdanau et al. for machine translation, who proposed to make a sequence rather than one vector. They also introduced an attention mechanism that learns to associate elements of that sequence with elements of the output. The book refers to section 12.4.5.1 for details.
A bottleneck argument makes the limitation concrete. The statement below is added here and is not printed in the book. Suppose the input has binary steps and the context has numbers, each taking one of values. There are different inputs and at most different contexts.
Click any coloured symbol to see what it means.
Predict first. With attention, what does each decoder step read from the input?
Choose an answer to see the result.
Inputs sharing one context
Challenge +8 XPSet the input to 4 steps, the context size to 2 and the values to 2 per number. At least one context must then be shared.
The second demo is the attention idea in its simplest form. The weights below are a toy, chosen so that each decoder step favors the input position it is aimed at. The book does not give this weighting, so the formula is added here and is not printed in the book.
Click any coloured symbol to see what it means.
Weights over the input positions
Challenge +8 XPSet the sharpness so that one input position takes more than 0.9 of the attention.
- An encoder-decoder pair reads an input of one length and writes an output of another, joined by a context.
- The last state of the encoder is the usual context, and the decoder is conditioned on it.
- A context of fixed size can hold only so many distinct inputs, so long inputs lose information.
- Attention lets each decoder step weight the input positions differently, which relaxes that limit.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.4 (pages 396–398), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.