Noise Robustness

Noise on the inputs of a linear model

Think of a marksman whose sight wobbles a little. Aim too hard at one spot and the wobble costs you. A marksman who learns to aim for the middle of the wobble will hit more often, and that aim is a little less extreme than the perfect aim. Noise on the input of a model does the same: a model that must work under wobbly inputs stops relying on large weights.

The book notes that for some models, adding noise with a very small variance to the input is equivalent to a penalty on the norm of the weights. A linear model with no bias has the form in one dimension. The demo below takes its simplest case, with one input and target .

Predict first. As the noise variance grows, what happens to the best weight ?

Choose an answer, then move the noise variance.

The equation on the left is the expected squared error when the input is , where is random noise of variance . The right side shows the expansion. This expansion is added here and is not printed in the book. The extra term is a penalty on the size of the weight.

The lowest point of the curve is the best weight, . This minimiser is added here and is not printed in the book. More noise moves it towards zero, so the noise has the same effect as a penalty on the weight. The penalty with the same strength would be written , and the noise variance plays the role of its coefficient.

Checkpoint. If the noise variance is 3, what is the best weight in this one-dimensional model?

Answer, then check the demo.

Noise on the weights

Noise can also be added to the weights. The book says this has been used mainly for recurrent networks, and it can be read as a stochastic form of Bayesian inference, in which the weights are uncertain. For a regression with the squared error, the book writes the objective as an expectation over the data and the noise:

The book perturbs the weights with a random term at each input presentation, and writes the objective with the noise included. The expansion of its square is Equation 7.32:

For small , the book states that minimizing the loss with added weight noise is equivalent to minimizing the loss with an extra regularization term. The book prints that term without a number:

The term pushes the parameters towards regions where small changes in the weights have little effect on the output. The book adds that for the simplified case of linear regression, this term collapses to a constant, which does not depend on the parameters.

Predict first. Take a linear model with a squared loss, and add weight noise to it. Does the best weight move?

Choose an answer, then raise the weight noise.

The demo uses the input , the target , and the weight . The expected loss is added here and is not printed in the book. The constant shifts the curve up and does not move its lowest point.

Checkpoint. Why does weight noise leave the best weight of a linear model unchanged?

Answer, then check the demo.

Label smoothing

Most datasets have some mistakes in their labels. The book warns that it can be harmful to maximize when the label is a mistake. One remedy is to model the noise on the labels explicitly. If the label is correct with probability and any other label may be correct otherwise, then the cost can be written with soft targets.

Think of a teacher who grades a test. Instead of writing 100 out of 100 on every paper, the teacher leaves room for a small chance that the answer key is wrong. The grade is still a clear pass, but no paper is treated as certainly perfect.

The book describes label smoothing for a softmax with outputs. It replaces the hard targets of 0 and 1 with targets of and :

The demo shows the two sets of targets for four classes, with . The smoothed target for the correct class is , and each other class gets . The numerical values in the readout are added here and are not printed in the book.

Predict first. With hard targets of 0 and 1, and a softmax classifier that keeps learning, what does its best prediction for the correct class look like?

Choose an answer, then move the smoothing.

The book explains the reason: a softmax can never predict a probability of exactly 0 or 1, so with hard targets it keeps learning larger and larger weights. Label smoothing avoids that pursuit of hard probabilities without discouraging the correct class. The book also notes that weight decay can prevent the same problem.

Checkpoint. With and four classes, what is the target for each wrong class?

Answer, then check the demo.

Noise and the penalty side by side

Here is the first result in two dimensions. Take one data point with input and target . Add noise of variance to each input entry, and compute the expected squared error. This expectation is added here and is not printed in the book. It equals the ordinary squared error plus , which is the penalty of the same strength.

Think of two ways to stop a shelf from sagging: buy a stiffer shelf, or put less on it. For a linear model with a squared error, the noise and the penalty have the same effect on the weights.

The book states that, for some models, input noise of infinitesimal variance is equivalent to a norm penalty. The demo checks this for one data point by computing the solution two ways: by averaging the squared error over the noise and minimizing it, and with the penalty of the same strength.

Checkpoint. For this data point, which quantity plays the role of the penalty strength?

Answer, then check the readout.

• For a linear model, input noise of small variance acts like a penalty on the weights, and the variance plays the role of the penalty strength.
• Weight noise on a linear model with a squared loss adds a constant and leaves the best weight where it was.
• Label smoothing replaces the targets 0 and 1 with and , so the model never has to reach a probability of exactly one.
• Noise can be a stronger regulariser than shrinking the parameters, and the book notes this is especially true when the noise is added to hidden units.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.5 (pages 242–243), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.