Parameter Norm Penalties

A penalty as a price on size

A model learns by lowering a loss. Left alone, the loss accepts any parameters that fit the training data, even very large ones. A parameter norm penalty adds a price for size to the loss. Training then minimizes a new objective, , which balances fit against size:

The first term is the ordinary loss . The second term is the penalty multiplied by the penalty strength . The strength is a number you choose, from 0 upwards. At the penalty vanishes, and each larger value makes size more expensive.

Think of cooking a stew. The loss measures how far the taste is from what you want. The penalty is the price of salt, and is what each unit of salt costs. With a price of zero you salt until the taste is right. With a higher price you stop earlier, with less salt and a slightly different taste.

The demo below uses one weight and one curve for the loss, so that you can see the balance directly. The loss has its bottom at = 2, which is the best weight with no penalty.

Predict first. As grows, where does the best weight go?

Choose an answer, then move the penalty strength.

The dashed curve is the loss. The violet curve is the price, , a bowl with its bottom at zero. The solid teal curve is their sum, and its lowest point is the best weight with the penalty, . The price pulls that point towards zero and the loss pulls it back towards .

The single weight has a curvature , which says how steeply the loss rises away from . The balance point is . This one-weight formula is the one-dimensional case of Equation 7.10, which is derived in Section 3 of this page.

Weight decay in one step

For neural networks the penalty usually touches the weights only, and the biases are left out. Each weight says how two variables interact, so fitting it well needs examples that show both variables in many conditions. Each bias controls a single variable and typically needs less data. Leaving the biases unpenalized adds little variance, while penalizing them can cause a significant amount of underfitting. The vector collects the penalized weights, and collects all parameters, weights and biases alike. The book also allows each layer to have its own , although searching for many values is expensive, so one shared value is common.

This penalty is the L2 penalty, also called weight decay. Other fields call the same method ridge regression or Tikhonov regularization. The penalty may also pull towards a point other than zero. Zero is the usual target, because it is the right choice when the sign of a weight is unknown.

For this section we take the simplest case, with no biases, so that equals . The penalty is . Adding it to the loss gives Equations 7.2 and 7.3:

A gradient step with learning rate is Equation 7.4. It steps against the penalty gradient, , and against the loss gradient together:

Rearranged, the same step reads as Equation 7.5. Before the usual gradient step, every weight is multiplied by , a number a little below 1. This is why the method is called weight decay: the weights decay towards zero on every step, even when the data give no gradient.

The demo below runs this step. It starts at , and the best weight with no penalty is 2. Switch off the data gradient to see the decay alone.

Predict first. The data gradient is zero. How much does one step shrink the weight when and ?

Choose an answer, then run the steps.

With the data gradient on, the weight moves towards and the gap to it is multiplied by on each step. The gap is multiplied by a factor that depends only on . While that number lies between 0 and 1, the gap shrinks without changing sign. Between 1 and 2 the gap changes sign on each step, so the weight overshoots the target and then settles. Past 2 the gap grows, and the weight diverges. A step that is too large is one of the first things to check when weights blow up.

Shrinking along curvature directions

Real losses have many weights. Near their best point a smooth loss looks like a bowl. Its shape is given by the quadratic approximation of Equation 7.6, where is the Hessian, the matrix of second derivatives of the loss at :

The bowl has no first-order term, since is a minimum where the gradient vanishes. Because it is a minimum, the Hessian is positive semidefinite, so the bowl never curves downwards. Its gradient is Equation 7.7, which is zero at :

Add the penalty gradient to that expression and solve for the new best point . Equations 7.8, 7.9 and 7.10 do this in three lines:

The Hessian is symmetric, so it has a set of perpendicular unit directions, its eigenvectors, which are the columns of . Along each eigenvector it has an eigenvalue , the curvature of the bowl in that direction. Write , and substitute into Equation 7.10, step by step:

The effect of weight decay is to rescale along the eigenvectors. The component along the -th eigenvector is multiplied by . A direction with large curvature keeps almost all of its weight. A direction with small curvature loses most of it. The book illustrates this in its Figure 7.1, where the loss is a flat valley in one direction and a steep wall in the other.

The dashed violet circles are the penalty, and the solid ellipses are the loss. The best penalized weight is where the two pulls balance. Tilting the valley turns the ellipses, but each direction still keeps its own share of the weight. The percentages in the readout are the factors for the two eigenvalues.

The book states that components with are barely affected, while components with are shrunk to almost zero. The demo lets you read that off directly.

Linear regression with weight decay

For linear regression the loss is exactly quadratic, so the same analysis gives an answer in terms of the data. The loss is the sum of squared errors, Equation 7.14. Here is the design matrix and the vector of targets:

With the weight decay term added, the book writes the objective as Equation 7.15.

The gradient set to zero gives the ordinary solution, Equation 7.16, and the ridge solution, Equation 7.17:

The matrix is proportional to the covariance matrix of the features. Equation 7.17 replaces it with the same matrix plus on the diagonal. The diagonal entries describe how much each feature varies, so the penalty makes the learner perceive every feature as more variable than it is. It then shrinks the weights on features whose link to the target is weak compared with that added variance. The new matrix is always invertible for .

Predict first. The demo below fits a two-feature regression. What happens to both weights as the penalty strength grows without limit?

Choose an answer, then move the strength.

The 24 training examples are generated from a seeded function, so the same data appear every time. The true relation is roughly with some noise. Raise the correlation to make the two features nearly the same column. Without the penalty, the weights then depend strongly on the noise, and the penalty holds them in place. With correlated features the path can be surprising: at ρ = 0.8 the second weight rises before both fall to zero.

The L1 penalty and its gradient

Another way to penalize size is the L1 penalty, the sum of absolute values of the weights, Equation 7.18:

With it, the regularized objective is Equation 7.19, and its gradient, which is a sub-gradient at zero, is Equation 7.20:

The penalty part of that gradient is . The function returns 1 for a positive weight, −1 for a negative one, and 0 at zero. Its size does not depend on how large the weight is. The L2 pull, , fades as the weight shrinks. The L1 pull stays at all the way to zero. At zero the absolute value has a corner, so the gradient there is a sub-gradient: any value between and is allowed. This corner treatment is added here and is not printed in the book.

The demo compares the two pulls, and then walks one weight under the penalty alone, with the data gradient removed. The walk starts at .

Predict first. From , with only the L2 penalty acting, what does the weight do after many steps?

Choose an answer, then run the walk.

The L2 walk multiplies the weight by the same factor on each step, so it shrinks by a fixed share of what remains. The L1 walk moves by the same fixed amount on each step. Once the L1 weight is that close to zero, it cannot leave the band of width around zero. Plain steps on the L1 penalty do not land exactly on zero. The next section shows how the L1 solution can contain exact zeros.

At a weight of 0.2, the L2 pull is and the L1 pull is . The L1 pull is five times stronger there, which is why it pushes small weights hardest.

Sparsity and the soft threshold

Take the quadratic approximation again, this time with the L1 penalty. The L1 version of the approximation is Equations 7.21 and 7.22. The book assumes the Hessian is diagonal, , which holds when the inputs have been decorrelated, for example with PCA. A general Hessian gives the L1 penalty no clean algebraic solution, which is why the book makes this assumption. With that assumption the problem separates into one small problem per weight:

Each small problem has a closed-form minimiser, the soft threshold, Equation 7.23. It moves the weight towards zero by and stops at zero:

Two cases follow from the formula. If the best unpenalized weight is at most in size, the penalty drives the weight exactly to zero. If it is larger, the weight moves towards zero by exactly and stays nonzero. The L2 solution in the diagonal case, from Equation 7.13, is times the unpenalized weight, and it is never exactly zero unless the unpenalized weight already is. This is the difference between the two penalties: L1 can give sparse weights, and L2 does not.

Sparsity makes L1 a tool for feature selection. A weight that ends at exactly zero suggests a feature that can be dropped. The LASSO combines this penalty with a linear model and a least squares loss.

Predict first. With and , how many of the eight weights in the bank below are exactly zero under the L2 penalty?

Choose an answer, then set the sliders.

The eight weights are fixed at 2.4, −1.6, 1.1, −0.9, 0.5, −0.25, 0.12 and −0.08. The threshold is , so the readout counts the weights that the L1 penalty sets to zero.

The Laplace prior

Many penalties can be read as a prior belief about the weights. Maximum a posteriori (MAP) estimation picks the parameters with the highest posterior probability, which is the likelihood times the prior. The logarithm turns that product into a sum, so the prior adds one term to the objective. The book develops this link in its Section 5.6.1, and the key steps are repeated here.

A Gaussian prior on each weight gives the L2 penalty. A Laplace prior gives the L1 penalty. Equation 7.24 writes the log prior for the L1 case, where the Laplace scale is :

The constant terms do not depend on the weights, so they do not change where the maximum lies. The part that matters is , the L1 penalty with its sign flipped, since the objective is maximized.

Predict first. Near , which of the two log priors drops faster?

Choose an answer, then move the strength.

The L2 curve is the log of a Gaussian with variance . The L1 curve is the log of a Laplace distribution with scale . Both are shown up to additive constants, so both are zero at .

The two curves cross at zero and at . The Laplace curve has a sharp tip at zero, while the Gaussian curve is smooth there. The soft threshold in Section 6 exploits that tip: it is the feature that sends small weights exactly to zero.

• The penalized objective adds a price to the loss. Training minimizes the sum, so the best weights move towards zero.
• L2 weight decay multiplies each weight by before every gradient step. Along each eigenvector of the Hessian, the best weight keeps the share .
• For linear regression, the penalty adds to the diagonal of , which makes the solution invertible for any .
• L1 weight decay applies a constant pull . In the quadratic picture it gives the soft threshold, which sets weights with small unpenalized values exactly to zero.
• The Laplace prior is the L1 penalty viewed as MAP inference. The Gaussian prior is the L2 penalty.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.1 (pages 230–236), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.