Gradient Clipping and Information Flow

Cliffs in the loss surface

Picture a path that is gently sloped for most of its length and then drops off a cliff. Walking downhill is easy until one step lands on the cliff face, where the step is far too long. A recurrent network over many steps has the same shape. Its loss is flat for most settings of its weights, and it falls sharply at a few. The book explains that such a cliff is exponentially steep in the number of time steps, because the weight matrix is multiplied by itself once for each step.

The book describes this landscape in Figure 10.17, which shows a loss with two parameters and a steep cliff face. The first demo uses a one-weight loss with the same kind of cliff. This loss is added here and is not printed in the book:

The slope of the loss is the gradient . The exponent is 20 here, so the loss changes slowly for small weights and very quickly once passes 1.

Predict first. The gradient at has size about 2.4. How large is it at ?

Pick an answer, then move the weight to see the gradient.

The gradient grows faster than the loss does near the edge. Between 0.90 and 1.05 the exponent makes the slope grow by a large factor, and an update that is correct on the flat part can throw the weight far up the face.

Second derivatives and steepness

The book notes an idea by Martens and Sutskever (2011). Second derivatives may vanish at the same time as first derivatives. If the second derivative shrinks at about the same rate as the first, the ratio of the two stays roughly constant. Second-order methods divide the gradient by curvature, so this ratio is what they rely on. Second-order methods have drawbacks: a high computational cost, a need for large minibatches, and a tendency to be attracted to saddle points. Sutskever et al. (2013) found that Nesterov momentum with careful initialization could achieve similar results. The book also notes that both approaches have largely been replaced by SGD applied to LSTMs.

An exponential shows the idea in its simplest form. Take the function below, with steepness and input . This example is added here and is not printed in the book.

The ratio of the slope to the curvature is

The ratio is the same at every point of the curve. A steeper curve, with a larger , gives a smaller ratio. In this example the curvature shrinks with the slope, so the step length that a second-order method takes stays the same along the curve.

Clipping the gradient

Clipping is a simple remedy that practitioners used for many years. The book gives two versions. Clipping the norm of the gradient is the one this section follows. If the norm is larger than a threshold , the gradient is rescaled to length before the update:

The update keeps the direction of the gradient, and its length is capped at . The book notes that this guarantees each step is still in the gradient direction, and that the update vector has a bounded norm.

Figure 10.17 shows the effect near a cliff in a network with two parameters. The demo below rebuilds the picture with a two-parameter loss. This loss is added here and is not printed in the book, and the book's own figure comes from a real network:

The valley floor is the curve . The shading shows the size of the gradient, and darker cells are steeper. Both runs start at and .

With the default step size of 0.05, plain descent lands on the cliff face, receives a very large gradient, and jumps far to the left onto the flat part. Raise the step size to 0.1, and plain descent leaves the picture at step 2. Clipped descent with threshold 1 still stays inside for step sizes up to about 0.3. It climbs the face in limited steps and keeps moving towards the valley rather than jumping across it.

Element-wise and norm clipping

The book describes a second version that cuts each entry of the gradient on its own, which is element-wise clipping. Experiments suggest that both forms work similarly. The book says that the norm version keeps the update aligned with the gradient, while element-wise clipping keeps only a descent direction. This section uses a two-entry gradient . The example is added here and is not printed in the book.

Predict first. Which kind of clipping keeps the direction of the gradient?

Pick an answer, then lower the threshold to test it.

Norm clipping multiplies both entries by the same factor, so the direction stays. Element-wise clipping cuts each entry separately. When the entries are unequal, cutting both by the same limit changes their ratio, so the direction changes. The book adds two caveats. Clipping the norm per minibatch is not the same as clipping the norm of the average over all examples, because examples with large gradients lose weight in the final direction. And when the gradient is infinite or not a number, the book suggests a random step of size v, which tends to work almost as well as clipping.

Regularizing for information flow

Clipping helps with exploding gradients, but it does not help with vanishing ones. The book describes a second idea: the gradient that flows back through the states should keep its size. Ideally the gradient after one step back through time has about the same size as before:

to be as large as

The book writes these as vectors. This module takes norms, as the regularizer below does, so the norm form is added here and is not printed in the book. Pascanu et al. (2013) turn the goal into a penalty:

Each term compares the stretch that one step applies to the gradient with the gradient's own size. The book says the gradient is treated as a constant for this penalty, so it does not need to be back-propagated. The demo below shows one term of the sum, for a single step. The gradient is fixed at , and you set the Jacobian by its four entries. The book adds that this regularizer is not as effective as the LSTM where data is abundant, such as language modeling.

The penalty reads only the first row of the Jacobian here, because the gradient points along the first axis. Other rows can change freely. A penalty of zero therefore does not mean that the Jacobian is a rotation.

• On a cliff, one gradient step can be far too long, so the update must be capped.
• Norm clipping rescales a gradient to length when its norm is larger than , and it keeps the direction.
• Element-wise clipping cuts each entry on its own, so it can change the direction of the update.
• The regularizer in Eq. 10.52 encourages the gradient to keep its size across steps, and the book reports that it works best with norm clipping, which the book calls particularly important for it, since without clipping gradient explosion prevents learning from succeeding.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.11 (pages 413–416), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.