Momentum

Before you start

This lesson uses one idea from calculus: the gradient of a loss tells you, for every parameter, which way the loss grows fastest. Moving the parameters the opposite way lowers the loss. Training a network repeats that move many times.

Plain stochastic gradient descent (SGD) does it like this. It takes a small random batch of examples, the minibatch of size , computes the average loss on it, and finds the gradient of that average:

Then it steps against the gradient, scaled by the learning rate :

Momentum changes only how that step is chosen. Everything on this page is built from the two lines above, so it is worth being sure of them. Two habits will help throughout.

The book writes parameters, gradients and velocities as vectors. Every operation here acts on each coordinate in the same way, so the demos use one or two coordinates without losing anything.

The problem momentum solves

SGD is popular, but it can be slow. The method of momentum, due to Polyak (1964), is designed to speed learning up in three situations that the book names:

1. High curvature. The loss is steep in some directions and nearly flat in others. This is demonstrated right below.
2. Small but consistent gradients. The slope is gentle, but it always points the same way. See Section 4.
3. Noisy gradients. Each minibatch gives a slightly different gradient. See Section 3.

The idea is short. Keep an exponentially decaying moving average of past gradients, and keep moving in the direction of that average. Here is the effect, rebuilt from the book’s Figure 8.5.

The contour lines show a loss that is a long, narrow valley with steep sides, which is what a poorly conditioned Hessian looks like. The valley is drawn lying flat, and the vertical direction is stretched so that you can see the zigzag. Click the map to start both methods from a new point. The orange arrows are the steps plain gradient descent would take from each point the momentum path visits. They point mostly across the valley, because that is where the slope is steepest. Gradient descent therefore spends its steps bouncing between the walls. Momentum cancels the back-and-forth parts of successive gradients, and the small steady part along the valley adds up. It does not remove the swinging completely, since the velocity overshoots, but the swings fade while the progress along the valley keeps building. The book compares this with its Figure 4.6, which shows gradient descent without momentum. Orange is that picture.

Try three things. Make the valley narrower and watch the gap grow. Set to 0 and notice that the violet path becomes the orange one. Raise to 0.99 and see momentum overshoot and circle in. is a hyperparameter you choose, and the best value depends on the problem.

The update rule

Formally, momentum adds one new variable . It plays the role of velocity: the direction and speed at which the parameters currently move through parameter space. It has the same shape as , one number per parameter. The update has two lines:

Read them in order, left to right.

• First line. Keep a fraction of the old velocity. Then subtract times the new minibatch gradient. The velocity is therefore an exponentially decaying average of the negative gradient: every old gradient is still inside it, each one weaker than the one after it.
• Second line. Move the parameters by the new velocity. Notice that the gradient no longer moves the parameters directly. It changes the velocity, and the velocity moves the parameters.
• The coefficient lies in [0, 1). It decides how quickly old gradients fade. At nothing is kept and the rule is plain SGD. As approaches 1, old gradients last longer.
• The name. It comes from physics, where the negative gradient is a force pushing a particle through parameter space. Physical momentum is mass times velocity. We take the mass to be 1, so the velocity vector is also the momentum of the particle.
• The ratio. The larger is compared with , the more the earlier gradients influence the current direction: a big keeps the old velocity loud, and a big lets the newest gradient shout over it.

The velocity accumulates exactly the gradient from Section 0, the of the minibatch average. Section 5 runs these two lines on actual numbers, one at a time.

What the velocity remembers

To see what “an exponentially decaying average of past gradients” means, apply the first line over and over. After one step the velocity holds the newest gradient. After two, it holds the newest gradient plus times the one before. Writing out step gives

Here is the gradient at the latest step and the one from steps ago. Each older gradient is multiplied by one more factor of , so its weight is . The first term shows that the starting velocity also fades by the same rule. Move below to see the weights.

The weights add up to a fixed number, because a geometric series has a closed form:

Two consequences matter. First, the average reaches back roughly steps, so remembers about the last 10 gradients and about the last 100. Second, the total weight is exactly the factor that appears in the speed limit of Section 4.

Why this helps with noisy gradients

Each minibatch gives the true gradient plus random error. Adding up many consecutive gradients keeps the part they share and lets the random parts partly cancel. The next demo feeds momentum a gradient that is truly 1 but arrives with noise, and shows the average that momentum follows. The violet curve is the negative velocity rescaled by so that it sits on the scale of a single gradient.

The price of a longer memory is a slower reaction. If the true gradient suddenly changed, an average over the last 100 steps would take on the order of 100 steps to follow. That is why is a choice and not simply “as large as possible”.

Step size and the speed limit

In plain SGD the size of a step is the length of the gradient times the learning rate, and nothing else. With momentum the size of a step depends on how large and how aligned a sequence of gradients is. Aligned gradients add up, and opposed gradients cancel. The step is largest when many successive gradients point in exactly the same direction.

The terminal velocity

Suppose momentum always sees the same gradient . The velocity grows until a step no longer changes it. Call that value . It must satisfy the update with on both sides:

Collect the terms on the left and divide by :

The particle accelerates in the direction of and stops accelerating at this terminal velocity. Its length, the size of each step, is the book’s Equation 8.17:

It helps to think of the momentum hyperparameter through the number . For it equals 10, so momentum multiplies the maximum speed by 10 compared with plain gradient descent. The common values of in practice are 0.5, 0.9 and 0.99, which give 2, 10 and 100.

The vertical axis counts step size in units of one plain-SGD step. Gradient descent stays at 1. Momentum climbs towards the limit, but not instantly. After steps it has covered about 63% of the way there (the step size after steps is units), so a very large takes a long time to build speed. This is the second of the three situations from Section 1: a small but consistent gradient is exactly what momentum turns into a long, fast stride.

Aligned or opposed

Now control the gradients yourself. Each press applies one unit gradient. Push the same way repeatedly and the velocity climbs to the limit. Alternate the direction and the pushes cancel.

With a constant gradient the speed approaches . With a gradient that flips sign every step, the velocity settles into a small swing of about and goes nowhere. This is the first situation again: across a narrow valley the gradient flips sign at every step, so momentum cancels it, while along the valley the gradient keeps its sign, so momentum builds speed.

Choosing and changing α

Like the learning rate, can be adapted during training. It typically begins with a small value and is raised later. The book adds that it is less important to adapt over time than to shrink over time. The ravine demo in Section 1 uses a fixed , so try changing it by hand there to see what an early small value and a later large value would do.

Algorithm 8.2, line by line

Here is the book’s Algorithm 8.2, SGD with momentum. Press Next line to execute it. The demo trains a single parameter on the loss , whose gradient is simply . A real minibatch would add noise to that gradient, and nothing else changes.

The last column runs plain SGD with the same learning rate on the same loss. The first step is identical, because the velocity starts at zero. After that, momentum pulls ahead as the velocity builds, and with the default numbers it passes the minimum at during iteration 4, where goes from 0.51 to −0.72. It does not stop there. It carries its velocity past the minimum, overshoots to about −1.9, swings back, and slowly settles. In this one-dimensional bowl there is no valley to cross, so plain SGD is simply smoother. Use the demo to read the arithmetic of Algorithm 8.2, not to judge speed. The valley in Section 1 is where momentum earns its keep.

The physical picture

Momentum can be viewed as simulating a particle that moves under continuous-time Newtonian dynamics. The picture builds intuition for how both momentum and plain gradient descent behave.

The position of the particle at time is . A net force acts on it, and with unit mass the force equals the acceleration:

This is a second-order differential equation in the position. We can make it first-order by introducing the velocity as its own variable:

Momentum then amounts to solving these equations by numerical simulation. The simplest numerical method is Euler’s method: take small finite steps, each one in the direction the current slope indicates. That explains the shape of the momentum update. It does not yet say what the forces are.

Two forces

The first force is proportional to the negative gradient of the cost, . It pushes the particle downhill. Plain gradient descent takes one step per gradient. The Newtonian version uses the same force to change the particle’s velocity instead. Think of a hockey puck on an icy surface: it gains speed on a steep slope and keeps sliding in that direction until the surface rises again.

A second force is needed. If gravity-like pull were the only force, the puck would slide down one side of a valley and up the other, and oscillate forever on perfectly frictionless ice. We add a force proportional to . In physics this is viscous drag, as if the particle were pushing through syrup. It drains energy until the particle comes to rest at a minimum. With both forces the net force is

(The book states the two forces in words, and this line is their sum.) Why viscous drag in particular? Part of the reason is convenience: an integer power of the velocity is easy to work with. But nature offers other integer powers, and the book rejects them one by one:

• Turbulent drag, proportional to the square of the velocity, as for a body moving through air. It becomes very weak at low speed, so it cannot bring the particle to rest. With only this force and a non-zero starting velocity, the particle drifts away forever, its distance growing like . So we need a lower power of the velocity.
• Dry friction, which is a power of zero: a force of constant size. This one is too strong. When the pull of the cost is small but not zero, the constant friction can stop the particle before it reaches a minimum.
• Viscous drag avoids both failures. It is weak enough for even a small gradient to keep the particle moving until a minimum, and strong enough to stop motion that the gradient does not justify.

Compare all four kinds of drag yourself.

In the valley (the loss ), the frictionless particle swings forever, viscous drag settles it exactly at the bottom, turbulent drag leaves it still swinging for a very long time, and dry friction parks it somewhere inside the band where the pull is smaller than the friction (at the default strength this is clearly short of the bottom, and a weaker friction narrows the band). In the no-valley panel the frictionless particle flies off at constant speed forever, the particle under turbulent drag keeps creeping away without ever stopping, while viscous drag and dry friction both bring it to rest.

From the physics back to the update rule

Take the viscous system and simulate it with Euler’s method using time steps of size , updating the velocity first and then the position. The velocity update is . Multiply the velocity by to measure it as distance per step, and this is exactly Equations 8.15 and 8.16 with

(This identification follows from the equations above and is not printed in the book.) It means that measures how much drag acts during one step, and that is the square of the step in time. The next demo checks it: the exact physics curve and the momentum update with the mapped values lie nearly on top of each other when is small, and drift apart as grows.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 8.3.2 (pages 296–300), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.