Dropout

Dropout as a cheap ensemble

Picture a football squad that must play any match. A coach who always names the same eleven trains only that team. A coach who benches a random set of players at each practice trains every subset to play well together. Dropout does the second thing to a neural network.

Dropout removes units from a network at random during training. A removed unit is multiplied by zero, so it contributes nothing to the forward pass. Each new example in a minibatch gets a fresh random set of removed units. The network that results is a large family of sub-networks that share their weights.

Section 7.11 showed that averaging several models helps, but training and running many large networks is expensive. The book describes dropout as an inexpensive approximation to training and evaluating a bagged ensemble of exponentially many neural networks.

The gray line rises by about 0.3 for every extra droppable unit on this base-10 logarithm scale, because the number of sub-networks doubles each time, and log₁₀ 2 is about 0.3. The teal line shows how many of them a training run of steps reaches. Each step samples one mask, so one step visits at most one sub-network. With large, almost every step meets a sub-network that no earlier step visited. The visit count is added here and is not printed in the book.

Sub-networks and their paths

Take the smallest network that shows the idea: two input units, and , two hidden units, and one output unit. Any of the four non-output units can be removed, which gives subsets. The book’s Figure 7.6 draws all sixteen. The demo below rebuilds them so that you can switch units on and off yourself.

A sub-network gives an output only if a path leads from an input to the output. The output reads only the hidden units, so a path needs at least one live input unit and at least one live hidden unit. The book notes that in this small example a large proportion of the sub-networks have no input or no path, and that this matters less for wider layers.

Predict first. How many of the sixteen sub-networks keep a path from input to output?

The count of nine is derived here and is not printed in the book. A unit is switched off by multiplying its output by zero. The book notes that radial basis function networks compare each unit’s state with a reference value, so the modification differs for them, and it says the same presentation works with other operations after a small change.

Training with random masks

Dropout training uses minibatch stochastic gradient descent, with one change. Each time an example enters a minibatch, sample a binary mask . It has one entry for every input and hidden unit. Each entry equals 1 with probability , the inclusion probability, and 0 otherwise. The entries are sampled independently. The book’s typical values are 0.8 for input units and 0.5 for hidden units.

The mask does not depend on the parameters or on the example’s values. The usual forward pass, backward pass and update then run on the thinned network. Figure 7.7 shows the forward pass: each unit is multiplied by its mask entry before it feeds the next layer.

Write for the cost of the network with parameters and mask . Dropout training minimizes the expected cost over masks:

This display restates the book’s sentence in symbols. It is not printed as an equation in the book. The expectation contains exponentially many terms, but sampling masks gives an unbiased estimate of its gradient.

Two things separate dropout from bagging. The sub-networks share parameters, so an exponential family of models fits in the memory of one network. Also, most sub-networks are never trained explicitly. The demo at the top of this page showed how few are visited. In the book’s words, a tiny fraction of the possible sub-networks are each trained for a single step, and parameter sharing causes the remaining sub-networks to arrive at good settings.

Averaging the ensemble

At test time the ensemble must give one prediction. For bagging, the book takes the arithmetic mean of the members’ probability distributions, as in Equation 7.52. For dropout, each sub-network is defined by a mask, so the mean runs over all masks, weighted by the probability with which the masks were sampled. That is Equation 7.53.

The sum over masks has exponentially many terms. The book says that 10 to 20 masks are often sufficient for good performance, but there is a faster route.

The book uses the geometric mean in place of the arithmetic mean. It cites Warde-Farley et al. (2014) for arguments and evidence that the geometric mean performs comparably. A geometric mean is not guaranteed to sum to one, so the book requires that none of the sub-models gives probability zero to any event, and then renormalizes:

Here is the number of units that may be dropped, and the root has order , so the result is the geometric mean over all masks.

Notation note. The book uses the letter d twice. In Equation 7.54 it is the number of units that may be dropped, and in Equation 7.57 it is a binary vector. This module keeps both meanings and writes the vector in bold, .

The book uses a uniform distribution over masks in Equations 7.54 and 7.55, so every mask gets the same weight, and it notes that non-uniform distributions are also possible. The demos on this page weight each mask by its training probability instead. That is a Bernoulli distribution in which each unit is kept with its inclusion probability, which is the non-uniform case. The two weightings give different exact rules. Uniform weights give one half for every input, and Bernoulli weights give each input its own inclusion probability.

The key idea is the weight scaling inference rule. Use one network with every unit present, and multiply the weights leaving each unit by the probability that the unit was kept. With inclusion probability one half, this divides the weights by two. The book says this approximates the ensemble with one forward pass. For a softmax classifier with input the calculation is exact, and it is set out in Equations 7.56 to 7.66.

Index the sub-models with a binary vector : element says whether input is kept, and the product zeroes the dropped inputs. Equation 7.57 gives one sub-model, and the ensemble then uses the normalized geometric mean over all binary vectors:

The book says the weight scaling rule is exact for many classes of models without nonlinear hidden units, including softmax regression, regression networks with conditionally normal outputs, and deep networks with hidden layers that have no nonlinearities. For deep nonlinear networks it is only an approximation. It often works well: Goodfellow et al. (2013a) found that the weight scaling approximation can classify better than Monte Carlo approximations, even when the Monte Carlo approximation may sample up to 1,000 sub-networks. Gal and Ghahramani (2015) found that some models do better with twenty Monte Carlo samples. The book concludes that the optimal choice of inference approximation is problem-dependent.

This derivation is added here and is not printed in the book. Weighting each mask by its Bernoulli probability, the sum over masks in the exponent averages each input to its own inclusion probability, because each unit is kept with that probability. So for a softmax classifier the exact rule with training weights multiplies input i by its inclusion probability. With the book’s uniform weights each input is multiplied by one half instead, and the two rules agree only when the inclusion probability is one half. The demo below uses one probability for both inputs.

Why the noise helps

The book reports that Srivastava et al. (2014) showed dropout to be more effective than other standard inexpensive regularizers, namely weight decay, filter norm constraints and sparse activity regularization. Dropout may also be combined with them to yield a further improvement. Two experiments in the book separate the reasons it works.

First, the randomness is not necessary for the regularizing effect. Fast dropout (Wang and Manning, 2013) derives analytical approximations to the marginalization. It converges faster, because the gradient is less noisy, and it nearly matches standard dropout on small problems, but it has not yet yielded a significant improvement on a large problem. Second, the same mask noise is not sufficient. Dropout boosting uses identical noise but trains the whole ensemble jointly to maximize the likelihood. It shows almost no regularization effect. The regularization effect of the bagged ensemble is achieved only when the sampled members are trained to perform well independently of each other.

Dropout also makes each hidden unit good in many contexts. The book compares this with sexual reproduction, which swaps genes between organisms and so favors genes that can be readily swapped. Each hidden unit must perform well regardless of which other hidden units are present in the model. Warde-Farley et al. (2014) also found that dropout improves generalization beyond what ensembles of independent models achieve.

The book also stresses where the masking noise acts. It is applied to hidden units, so it is a form of adaptive destruction of information rather than destruction of raw input values. If a hidden unit detects a face by finding a nose, dropping that unit erases the information that a nose is present, and the model must find the face another way. Unstructured noise added at the input would need to be so large that nearly all information in the image disappeared.

The book gives a reason for the masking noise to be multiplicative. If the noise on a rectified linear unit were additive with fixed scale, the unit could learn to become very large, which makes the added noise insignificant by comparison. Multiplicative noise does not allow that solution. The demo below compares the two kinds of noise on a single unit, so you can see how the signal to noise ratio behaves as the weight grows. This comparison is added here and is not printed in the book.

The book also notes that multiplying the weights by noise drawn from a normal distribution with mean 1 can outperform dropout based on binary masks. Because the mean is 1, the standard network approximates the ensemble without any weight scaling. For linear regression, dropout is equivalent to L2 weight decay with a separate coefficient for each input feature, and the size of each coefficient is set by that feature’s variance (Wager et al., 2013). For deep models this equivalence does not hold.

The book also notes that batch normalization reparametrizes a model in a way that adds both additive and multiplicative noise to the hidden units during training. Its primary purpose is to improve optimization, but the noise can have a regularizing effect and sometimes makes dropout unnecessary (Section 8.7.1).

Cost, capacity and limits

Dropout is cheap. Each training update needs only O(n) computation per example, to generate n random binary numbers and multiply them by the state. Inference costs the same per example as a network without dropout, after the weights are divided once before inference begins. Dropout does not significantly limit the type of model. It works with feedforward networks, restricted Boltzmann machines and recurrent networks (Bayer and Osendorfer, 2014; Pascanu et al., 2014a).

The cost of a complete system can still be significant. Dropout reduces the effective capacity of a model, so the model must be enlarged to offset this. The book says the optimal validation set error is typically much lower with dropout, but at the cost of a much larger model and many more iterations of the training algorithm. For very large datasets, regularization confers little reduction in generalization error, and the extra cost may outweigh the benefit.

Dropout is weaker when very few labeled examples are available. The book reports that Bayesian neural networks (Neal, 1996) outperform dropout on the Alternative Splicing Dataset, where fewer than 5,000 examples are available. When additional unlabeled data is available, unsupervised feature learning can gain an advantage over dropout.

The demo applies arithmetic that the book implies but does not write. If a layer keeps each unit with probability , then active units on average need a layer of units. The formula is added here and is not printed in the book.

Dropout is still the most widely used implicit ensemble method. DropConnect drops individual weights, and stochastic pooling randomizes the pooling of convolutional networks. The book notes that any random modification of a network can be used, provided the modification family is one that neural networks are able to learn to resist.

• Dropout trains an ensemble of up to sub-networks that share their parameters, at O(n) extra work per example per update.
• Inference averages the sub-networks with the weight scaling rule. For softmax it is exact, because the normalized geometric mean collapses to weights at inclusion probability one half.
• The masking noise acts on hidden units, so each unit must stay useful alongside many different companions.
• Dropout needs a larger model and more iterations, and it is weaker when labeled data is scarce.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.12 (pages 258–268), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.