Section 1
Repeated multiplication by one number
Start with the simplest case: a single number that is multiplied by itself again and again. Each step of a recurrent network without inputs and without a nonlinearity does the same thing with a matrix. The book writes that recurrence as Equation 10.36, and it calls it a very simple recurrent network that lacks a nonlinear activation function and lacks inputs.
Click any coloured symbol to see what it means.
The scalar case is the cleanest picture. After steps the state is times the start, and the size of decides what happens. Below 1 the product shrinks towards zero. Above 1 it grows without bound. The book stresses that this problem is particular to recurrent networks. A feedforward network with a different weight at each step behaves differently, and the book notes that very deep feedforward networks with carefully chosen scaling can avoid vanishing and exploding (citing Sussillo, 2014).
Predict, then reveal
Set to 0.9 and apply it 20 times. Predict the result before you press a button.
Choose an answer. The result appears after you choose.
Challenge +8 XPWith steps, make the value greater than 10 by choosing .
Where the book is briefThe book states the scalar case in one sentence and does not draw it. The chart above is an illustration written for this page, and it shows the value on a logarithmic axis so that small and large values both fit.
Section 2
Eigen directions and the power method
A matrix has a simpler picture when its eigenvectors are used. The book writes the state after steps as Equation 10.37, and it writes a matrix with an eigendecomposition as Equation 10.38.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Taking powers of is the whole effect. Each eigenvalue is raised to the power . The book says that eigenvalues below one in size decay to zero, eigenvalues above one explode, and any part of the start that is not aligned with the largest eigenvector is eventually discarded.
The book then simplifies further, with orthogonal , to Equation 10.39:
Click any coloured symbol to see what it means.
Inconsistency in the bookAs printed, (10.39) has the form . With (10.38) as printed, the consistent form is , which is what the demo below computes. The printed form matches the eigendecomposition , not the one in (10.38). For a rotation by 0.6 radians and = diag(1.02, 0.5), the printed form gives a second component of the opposite sign to the consistent form at . This note is added here and is not printed in the book.
The demo below uses a symmetric matrix, so the transpose in (10.36) and (10.37) changes nothing. The starting state is the unit vector along the first axis.
Two eigenvalues and a rotation
state after each stepeigen directions
Challenge +8 XPMake the length after 10 steps between 0.45 and 0.55 times the starting length. Keep the angle at 0.
Why a rotation keeps the eigen form intact
A rotation is orthogonal, so . The matrix then has the eigenvectors as its columns of , and the eigenvalues on the diagonal of . Taking the power of leaves the rotation in place and raises only the diagonal. This derivation is added here and is not printed in the book.
Many layers with fresh weights
The book compares the recurrent case with a feedforward network that uses a different weight at each step. The weights are generated randomly and independently, with zero mean and variance . The variance of the product is then of order , where is the number of layers.
The book then chooses the per-layer variance so that the product has a desired variance . In words, the per-layer variance is the th root of the target. The exact identity behind this is added here and is not printed in the book: for independent factors with zero mean, the second moment of the product is the product of the second moments.
Click any coloured symbol to see what it means.
Variance of a product of random layers
variance of the producttarget band
Challenge +8 XPUse 20 layers and choose so that the variance of the product is between 5e-7 and 2e-6.
Sampling is unreliable hereThe formula is an exact identity for the second moment, and the demo shows that exact value rather than a sample. Simulated products agree with it for a few layers. For about twenty layers the sample variance is unreliable: a few large products dominate it, and repeated samples of the same size can differ by more than an order of magnitude. This remark is added here and is not printed in the book.
Section 4
Composing a nonlinear map
Recurrent networks also compose a nonlinear map many times, once per step. The book shows the effect in Figure 10.15: a 100-dimensional state after each time step, projected onto one line, plotted against the coordinate of the start along a random direction. The book describes the result: most values have a tiny derivative, some have a large derivative, and the curve alternates between rising and falling many times.
The rebuild below uses a seeded random layer with 100 hidden units. Its curves differ in detail from the book’s figure, but they show the same effect. The gain scales the weight matrix. Watch how the slope at the center of the curve after 5 steps changes with the gain.
Click any coloured symbol to see what it means.
A random layer composed five times
after step 1step 2step 3step 4step 5
Challenge +8 XPTune the gain so that the size of the center slope after 5 steps is between 0.5 and 1.5.
What the book says about trainingThe book notes that gradients in a long chain tend to vanish most of the time and explode rarely, and that storing memories robustly pushes the network towards the region where gradients vanish. It cites Bengio et al. (1994), who found that the chance of successful training by stochastic gradient descent falls towards zero for sequences of length 10 or 20.
Summary
- A recurrent weight multiplies the state at every step, so its size decides whether the product vanishes or explodes.
- Eigenvalues below one in size decay, eigenvalues above one explode, and the largest eigenvalue dominates the long-run direction.
- Random layers scale the same way: the variance of a product of layers is , so a feedforward network avoids the problem only with a carefully chosen per-layer variance.
- Composing a nonlinear map many times produces steep and flat parts, and the book links this to the difficulty of learning long-term dependencies.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.7 (pages 401–403), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.