Echo State Networks

A pond that remembers

Picture a pond. A stone makes ripples that spread and fade over several seconds, and the surface still carries a trace of the stone long after it sank. An echo state network works in the same way. Its recurrent and input weights are set once, so that the hidden state echoes past inputs. Only a readout from the state is trained.

The book calls this idea reservoir computing, after the reservoir of temporal features formed by the hidden units. The recurrent and input weights are not learned. Only the output weights are, and the book notes that this makes the training criterion convex in those weights when the criterion is mean squared error.

Predict before you look. A reservoir at spectral radius 0.3 and another at 0.8 are each asked to recall the input from 10 steps ago. Which one recalls it better?

Reading the memory from the state

A readout is a linear map from the state to a prediction. To test memory, the readout predicts the input from steps ago. The score is the normalized mean squared error, . A score of 0 is a perfect prediction, and a score of 1 is no better than predicting the average of the target.

The readout weights are fitted by ridge regression, which is linear regression with a small penalty that keeps the solution stable. This fit is added here and is not printed in the book.

Forgetting and the edge of stability

The book says that the spectral radius of the Jacobian matters. It defines the spectral radius as the largest absolute value of an eigenvalue of the Jacobian, and the same definition holds for a matrix.

For a linear map, a difference along an eigenvector is multiplied by its eigenvalue at every step, so a spectral radius above one grows and one below one shrinks. The book notes that the derivative of the nonlinearity approaches zero on many steps, which helps prevent explosion. It also says that the most recent work on echo state networks advocates a spectral radius much larger than one, citing Yildiz et al. (2012) and Jaeger (2012).

Learning only the readout

With the states fixed, the training criterion depends only on the readout weights. For a linear readout and a mean squared error, the criterion is a quadratic bowl in those weights. The book says this is what makes the problem convex, so simple methods solve it reliably.

Here is the average outer product of two states, is the average product of each state with its target, and is the average of the squared target. These three averages are added here and are not printed in the book. The bowl has one lowest point, and the readout weights that reach it are the fitted weights.

Separation and non-normal matrices

Take two states that start close together and follow the same input. Their gap changes at the rate set by the transition, as in the perturbation argument of Section 8.2.5. For a pair of complex eigenvalues, the transition is a rotation with a scale, so the length of the gap is multiplied by exactly the same factor at every step. For a real pair the gap depends on its direction.

The book notes that neither the recurrent weights nor the Jacobian need be symmetric, so their eigenvalues can be complex, and that complex eigenvalues describe potentially oscillatory behavior. The demo below compares the two pairs for the same starting direction.

The book’s contractive wording refers to the long run, and a single step can behave differently when the matrix is not normal. Take a matrix with the same eigenvalue twice and a large entry above the diagonal. Its eigenvalues are both one half, so its spectral radius is one half, yet one step stretches some unit vectors a great deal.

Summary

• An echo state network fixes its recurrent and input weights and learns only a linear readout, which makes the training criterion convex.
• The memory of a reservoir, measured by , depends on the spectral radius and on the delay, and it falls as the delay grows.
• The book’s spectral radius is the largest eigenvalue in size. A radius below one forgets the past over many steps, and the book advocates radii above one for reservoirs.
• For a non-normal matrix, the spectral radius alone does not bound the stretch of a single step.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.8 (pages 404–405), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.