Parameter Initialization Strategies

Why initialization matters

Most training algorithms for deep models are iterative. They need a starting point, and the starting point matters a great deal. The book says the initial point can decide whether the algorithm converges at all, since some starting points are so unstable that training fails numerically. When it converges, the starting point affects how fast it converges and whether it reaches a point of high or low cost. Points of comparable cost can also have very different generalization error, so the choice of start affects that too.

The book admits that modern initialization strategies are simple and heuristic. Most are based on some nice property at the moment of initialization, and the book notes that it is not well understood which of those properties survive once learning begins. The one property known with complete certainty is symmetry breaking. Two hidden units with the same activation function and the same inputs must start with different parameters. If they start the same, a deterministic learning algorithm updates both in exactly the same way, and they stay identical.

The demo below uses a network with two hidden units on one input. The model is . The data are twenty points on the interval from minus one to one, with targets . Both units start with and . The second weight starts at , and the slider sets . With the units are identical, and the chart shows one curve.

Run the demo at zero difference and watch the two curves lie on top of each other at every step. The readout gives the gap at the chosen step, and it is exactly zero. This is the symmetry argument in one picture: identical units receive identical gradients, so the difference never appears. The derivation of the zero gap is added here and is not printed in the book, but it follows from the statement in the book. Now start the units apart. The gap grows, and the two units compute different functions.

The book goes on to say that random initialization is the natural way to make every unit compute a different function. It can also be cheaper than searching for a set of mutually different basis functions. For example, if there are at most as many outputs as inputs, Gram-Schmidt orthogonalization of an initial weight matrix guarantees that the units differ, but that costs computation. A random draw from a high-entropy distribution is cheaper, and it is unlikely to give two units the same function.

Random weights and their scale

The book says that biases are usually set to heuristically chosen constants, and that only the weights are drawn at random. Almost always the weights are drawn from a Gaussian or a uniform distribution. The choice of the two does not seem to matter much, but the scale of the distribution matters a great deal, for the outcome of optimization and for generalization.

Large initial weights give a stronger symmetry-breaking effect and help avoid losing signal in the linear part of each layer. The book notes that larger values in the matrix give larger outputs from the matrix multiplication. Weights that are too large can cause values to explode during forward or backward propagation. In recurrent networks, large weights can produce chaos. Large weights can also saturate the activation function, which removes the gradient through saturated units. The ideal scale balances these forces.

One heuristic samples the weights of a layer with inputs and outputs uniformly from . Glorot and Bengio (2010) suggest the normalized initialization of Equation 8.23:

The rule is a compromise between two goals: keeping the activation variance the same from layer to layer, and keeping the gradient variance the same. The book says it is derived for a chain of matrix multiplications with no nonlinearities, which real networks violate. It adds that many strategies designed for the linear model perform reasonably well on the nonlinear one. The demo below uses exactly that chain: twenty layers of width 100, with no nonlinearity. Each layer multiplies by a matrix with weights drawn from , and the chart shows the spread of the activations.

At a scale of , the spread shrinks by a constant factor at each layer and almost vanishes by the twentieth. At the Glorot scale, , the spread stays near one. At a larger scale it grows, and the readout shows the growth. The per-layer factor is added here and is not printed in the book: for a uniform weight on , the variance of each weight is , so the spread is multiplied by per layer.

Sparse initialization and the scale as a hyperparameter

The rule that gives every weight the same standard deviation, such as , has a drawback. Every individual weight becomes very small as the layers get large. Martens (2010) introduced sparse initialization instead. Each unit starts with exactly non-zero weights, which keeps the total input to the unit independent of the number of inputs , without making each weight shrink with . It gives more diversity among the units at the start.

The book warns that sparse initialization imposes a strong prior on the weights chosen to be large. It takes a long time for gradient descent to shrink wrong large values, so this scheme can cause trouble for maxout units, which have several filters that must be coordinated. The demo below compares a dense layer with a sparse one. Both are fed the same input, and the chart shows the spread of the units' pre-activations. The dense weights have variance . The sparse weights have non-zero Gaussian values of unit variance.

The sparse spread grows with the square root of , which is added here and is not printed in the book. The dense spread stays at one. The book's point is that the choice between dense and sparse can itself be a hyperparameter, and the same holds for the scale of each layer.

The book gives advice on choosing the scale. When computing resources allow it, treat the initial scale of each layer as a hyperparameter and choose it by a search, such as random search. Another rule of thumb is to look at the range or standard deviation of the activations on one minibatch. If the activations shrink as they move forward, increase the weights of the first layer that is too small, and repeat. The book notes that this procedure is cheaper than hyperparameter optimization on validation error, and that it has recently been specified more formally by Mishkin and Matas (2015).

Initialization as a prior

The perspectives of regularization and optimization give different advice. The optimization view says the weights should be large enough to pass information on. Regularization concerns push them smaller. Stochastic gradient descent makes small changes to the weights and tends to stop near the initial parameters, either because it is stuck in a region of low gradient or because an early stopping rule based on overfitting triggers. That expresses a prior that the final parameters should be close to the initial ones.

The book recalls that gradient descent with early stopping is equivalent to weight decay for some models. In the general case it is not the same, but it is a loose analogy for the effect of initialization. Placing the parameters at acts like imposing a Gaussian prior with mean . From that view it makes sense to choose near zero, because the prior then says units are more likely not to interact. If is set to large values, the prior specifies which units should interact and how.

The demo below uses a one-dimensional quadratic with minimum at , starting from , and gradient descent with step size . The parameter after steps is . This closed form is added here and is not printed in the book. Stopping after a few steps keeps the parameter near its starting value, and the readout shows how far it has moved.

The readout gives how far the parameter has moved from its start at the chosen step, and how far it still is from the minimum. Early stopping does not give the same answer as a penalty, but it does limit how far the parameter can move from the start, which is the sense in which initialization acts as a prior.

Biases and special parameters

Zero biases are compatible with most weight initialization schemes. The book lists three situations in which other values help. The first is an output bias. It is often useful to set it to match the marginal statistics of the output in the training set. If the initial weights are small, the output depends only on the bias, so the bias can be set to the inverse of the activation applied to those statistics. For a classifier with class frequencies , the book says to solve for the bias vector .

The demo below uses three classes with marginal frequencies . The first class has the share chosen by the slider, and the other two split the rest equally. Zero biases predict uniform probabilities. Biases set to the marginal make the prediction match the data. The solution of is up to a constant, which is added here and is not printed in the book.

The book notes that a ReLU hidden unit may get a bias of rather than zero, to avoid saturating it at initialization. That approach does not suit every scheme, such as random walk initialization. A unit can also gate another unit: a unit with output multiplies another output , and lets learn. The book recommends setting the forget-gate bias of an LSTM to one, following Jozefowicz et al. (2015).

Variance and precision parameters can usually be set to one safely. The book uses a linear regression with a conditional variance estimate as its example:

Here is a precision parameter. The book suggests an alternative: assume the initial weights are close enough to zero to ignore them, set the biases to match the marginal mean of the output, and set the variance parameters to the marginal variance of the output in the training set.

Initialization from other learning

Initialization can also come from learning. The book describes a common strategy from Part III: initialize a supervised model with the parameters of an unsupervised model trained on the same inputs. One can also train on a related supervised task. The book adds that even training on an unrelated task can sometimes give an initialization with faster convergence than a random one. These strategies may help because they encode information about the data distribution, or because they set the parameters to the right scale or make the units compute different functions.

The demo below makes the idea concrete on a one-dimensional quadratic with minimum at zero. A random start is far away, at five. A warm start is the solution of a related task, placed at the distance the slider sets from the target. Both use gradient descent with step size given by the slider. The readout gives the steps each needs to get within 0.01 of the target.

The book does not give a formula for how much a warm start helps. The book's reasons are that such a start may encode information about the data distribution, may set the right scale, or may make units compute different functions. The demo does not test those reasons. It shows only that a start closer to the target finishes sooner.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 8.4 (pages 301–306), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.