Gradient-Based Learning

Training by descent on a non-convex loss

Training a network is, at heart, the same as training any other model with gradient descent. Pick a cost, pick a way to represent the output, and move the parameters downhill. A linear model has a single valley, so any downhill route ends at the bottom. A neural network does not. Its loss has many valleys and ridges, because of the nonlinearity. The book calls such a loss non-convex.

A non-convex loss has no guarantee that the descent finds the global bottom. It depends on where the descent starts. For that reason the book says that the initial weights should be small random values, and that the biases may start at zero or at small positive values. The demo below uses a one-parameter loss with two valleys, so you can see the start decide the answer.

Predict before you move anything. Starting at the point marked by the default value, which valley does gradient descent reach?

Choose a valley to see the result.

The curve is a stand-in for a real loss, drawn in one dimension. Drag the start point along the horizontal axis on the picture, or use the slider. The path shows where each step lands. Near the bump at about 0.17 the slope points one way, so a start on either side ends in a different valley. The step size changes how the path settles. A large step makes it swing from side to side around the bottom of a valley, and a small step approaches the bottom smoothly. On this curve the step size does not change which valley wins, because the bump is the boundary between the two valleys.

Learning by maximum likelihood

Most modern networks learn a full conditional distribution , which is the probability of the output given the input . The model is , and the data come from the distribution of training pairs. Maximum likelihood chooses to make the training data as probable as possible. Turned into a loss to minimize, this is the cross-entropy between the data and the model:

Here is an average over the training pairs, and the minus sign turns a quantity to maximize into one to minimize. The form of depends on the model. The book’s example: if the model is a Gaussian with mean and unit variance, the negative log-likelihood is a squared error, up to a constant that does not depend on . The derivation is short, and the book prints the result:

The factor one half and the constant do not change where the minimum lies. Maximum likelihood with a Gaussian output is therefore the same as minimizing the squared error. The book adds that this holds whatever function predicts the mean, not only a linear one. The demo uses one training target, y = 1.3, and one prediction f that you move. The next demo checks the claim with a standard deviation that you can change. The book fixes the variance at 1. The demo does not, and the variance is what makes the comparison interesting.

The book also warns about the other side of this choice. A model that can control its own variance can make the likelihood of the training data very large by shrinking the variance towards zero, so the cross-entropy has no finite minimum. Chapter 7 describes regularization techniques that stop the model from gaining unlimited reward in this way.

Learning conditional statistics

Sometimes we do not want the whole distribution, only one statistic of given . Then we can pick a cost whose minimiser is that statistic. The book derives two results with the calculus of variations, which asks for the best function rather than the best parameter. You do not need that tool here. The two results are stated below, and the demo tests them on a small set of numbers.

If we minimize the mean squared error over all functions, the problem is

and its minimiser is the conditional mean of :

Minimizing the mean absolute error instead, the book shows that the minimiser is the conditional median. The result is stated as Equation 6.16, and the derivation is in the book. The next picture uses six numbers, so the mean and the median are easy to see.

Mean squared error and mean absolute error often lead to poor results with gradient-based optimization. Some output units that saturate give very small gradients when they are combined with these costs. This is one reason cross-entropy is more popular, even when only a statistic is wanted.

Predict first. For the six numbers in the demo, which constant minimizes the mean absolute error?

Choose an answer.

Linear units and the problem of saturation

The simplest output unit computes an affine function of the features, , with no nonlinearity. Its natural pairing is a Gaussian output with mean , equation 6.17:

Maximizing that likelihood minimizes the squared error, as Section 1 showed. A linear unit does not saturate, which means its output keeps changing in proportion to its input over the whole range. Its gradient never shrinks, so gradient descent has no trouble with it.

One recurring theme of neural network design is that the gradient of the cost must stay large and predictable enough to guide learning. A function saturates when it becomes very flat, so its slope, and therefore the gradient that flows through it, is almost zero. Saturation is the main threat to that gradient. The demo below measures the gradient for three pairings of output and cost.

Predict first. The model is confidently wrong: the pre-activation is and the target is 1. Which of the three gradients is the smallest in size?

Choose an answer.

The linear unit’s gradient grows with the error, which is large but stable. The sigmoid with squared error is the problem. Its gradient carries the factor , which is small at both ends of the curve. The sigmoid with cross-entropy has no such factor, as Section 4 shows, so its gradient stays large while the answer is wrong.

Sigmoid output units for binary targets

A binary target is 0 or 1, and the network must output , a number between 0 and 1. The book first tries the obvious route, a linear unit clipped to the interval, equation 6.18:

The demo below clips a linear value to the interval [0, 1]. Move it past either end and watch the slope.

Outside the interval the slope is zero, so gradient descent receives no signal about how to fix a wrong answer. This is why the book uses a sigmoid instead. The sigmoid output unit is

where is the logistic sigmoid. The sigmoid never reaches 0 or 1, so the output is always a valid probability. Its logit comes from the linear layer, and the book derives the form of the sigmoid by exponentiating and normalizing two unnormalized probabilities. The derivation has three steps. The log of the unnormalized probability is linear in and (6.20). Exponentiating gives the unnormalized probability (6.21), and dividing by the sum over the two values of gives a Bernoulli distribution (6.22). Written with as a function of , the result is 6.23:

The negative log-likelihood of one example is 6.24. Using the sigmoid form, it becomes 6.25, which the book then writes as a softplus, , in 6.26. The identities that make this simplification are the sigmoid properties of section 3.10 of the book.

The softplus form explains the behavior. The loss saturates only when is very negative, which is when the model already has the right answer. When has the wrong sign, the argument is for large values, and the gradient does not shrink. The next demo checks this for both targets.

Predict first. The model is confidently wrong: the target is 1 and the logit is . Which of these is closest to the true loss?

Choose an answer.

Softmax output units for classes

In the binary case the network outputs one number, , equation 6.27, and it predicts its logit . For a variable with classes, the network needs a vector of probabilities that are each between 0 and 1 and sum to 1. The book first has a linear layer produce unnormalized log probabilities, (6.28), and then applies the softmax, equation 6.29:

Exponentiating makes every value positive, and dividing by the sum makes them add to 1. The softmax is the multi-class version of the sigmoid. With two classes and one logit fixed at zero, it returns the same probability as a sigmoid.

The name is a little misleading. The softmax is a soft version of the arg max, not of the max. Taking the log of the softmax gives 6.30:

The first term, , pushes its own logit up and cannot saturate. The second term is roughly the largest logit, so the negative log-likelihood strongly penalizes the most active wrong answer.

The demo below has three logits. Change them and watch the probabilities. Then press the button that adds the same number to all three.

Adding the same number to every logit leaves the probabilities unchanged, equation 6.32. The book uses this to write a stable version, equation 6.33, which subtracts the largest logit first. The output depends only on how far each logit lies below the largest one. The softmax can saturate as well, when the differences between its inputs become extreme.

The book makes two more points. Under maximum likelihood the softmax is driven to match the observed frequency of each outcome, equation 6.31. Squared error is a poor choice here, because it can fail to change a confident wrong output.

Unregularized maximum likelihood drives the softmax towards these fractions. Limited capacity and imperfect optimization mean that the model only approximates them in practice.

Other output types and mixtures

The principle that works for the linear, sigmoid and softmax units also works for almost any output: write down a distribution and use its negative log-likelihood as the cost. The network outputs the parameters of that distribution. A Gaussian with unknown variance, for example, outputs its mean and its precision.

The book prefers precision, the inverse of variance, because a log-likelihood written in precision uses only multiplication, addition and a logarithm, which have gentle gradients. A positive precision is obtained from a raw activation with the softplus, so that is always positive. The demo shows that the raw value can be anything while the precision stays positive.

For several output dimensions the book uses a diagonal precision matrix, equation 6.34, which has one positive number per dimension.

When the output is a mixture, the network can represent several peaks for the same input. The book calls this multimodal regression and writes a Gaussian mixture with components, equation 6.35:

Three outputs are needed: the mixing weights through a softmax, the means , and the covariances. The mixing weights must be positive and sum to 1, and the covariances must be positive definite. A diagonal covariance keeps the determinant cheap. Full covariance matrices are rare for the same reason, because their determinant and inverse cost O(d³). Neural networks with Gaussian mixture outputs are called mixture density networks. Figure 6.4 of the book shows samples from one such network, with three mixture components and an input x that varies. The first demo below draws the density of a two-component mixture for one fixed input.

The sum of the two weighted components is a density that integrates to 1. When the means are far apart compared with the spread, the density has two peaks, and a single Gaussian could not follow it. This is why the book uses mixtures when the output has more than one plausible value.

The second demo rebuilds the picture of Figure 6.4 as a toy. The input is the horizontal position and the output is the vertical position. Each dot is a sample from one of three components, and the weights of the components change with the input. The weights and means are invented for this picture and are not the book’s numbers.

• Training is gradient descent on a non-convex loss, so the start point and the initial weights matter.
• Maximum likelihood gives the cost: cross-entropy for a distribution, and squared error for a Gaussian with fixed variance.
• Squared error learns the conditional mean, and absolute error learns the conditional median.
• A sigmoid with cross-entropy keeps a large gradient on a wrong answer. A sigmoid with squared error does not.
• A softmax gives one probability per class, and it depends only on the differences between the logits.
• For other outputs, write the distribution and let the network output its parameters. Precision is kept positive by softplus.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 6.2 (pages 177–190), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.