Regularized Autoencoders

Regularization instead of a small code

Section 14.1 limited the autoencoder by making its code small. This section keeps the code as large as you like, and limits the machine another way: a regularizer, an extra term or rule that prefers some solutions over others. Picture a sculptor asked to copy a statue. Without a rule she may copy every chisel mark. With a rule that favours simple, smooth shapes she must decide which marks matter. The rule is the regularizer.

An autoencoder whose code has more numbers than its input is called overcomplete. A linear one with an overcomplete code can copy its input exactly, so the copy loss alone teaches it nothing. Predict what happens when you train it on the copy loss only.

The book names three preferences that regularized autoencoders can add: sparsity of the code, small derivatives of the code, and robustness to noise or to missing inputs. Pick one below to see which equation in this section carries it.

A regularized autoencoder can be nonlinear and overcomplete, and still learn something useful even when its capacity is large enough to learn the identity. The book adds that many generative models with latent variables can also be read as autoencoders, and that they learn useful codes because they are trained to raise the probability of the data rather than to copy it.

Sparse Autoencoders

A sparse autoencoder adds a sparsity penalty () on the code to the reconstruction loss. The book writes the training criterion as:

Here is the reconstruction loss from section 14.1, and punishes the code for having many nonzero units. The strength of the penalty is , a small positive constant. A sparse code is useful for a second task such as classification, because only a few features are active for any input.

The clearest way to see sparsity is a one-number toy. Take the identity decoder, so the reconstruction is the code itself, and an absolute value penalty ||. The best code for an input then solves a small problem, and its answer is below. This formula is added here for the demo and is not printed in the book:

Read it as shrinkage. The code moves toward zero by /2. Once the input is within /2 of zero, the code is exactly zero. The penalty subtracts a fixed amount from the size of the code, so small values become exactly zero.

The second idea is about the prior. The book reads the sparse autoencoder as a generative model. A latent variable is a quantity the model infers rather than observes, and here it is the code . The model gives the code a prior, its probability () before it sees the input. A log turns products into sums, which is why the book works with log probabilities. The book replaces the sum over all codes by one highly likely code, so the log probability splits into a prior term and a likelihood term:

The sparsity penalty is the prior term in (14.4). The book takes the Laplace distribution as the prior. A density is a curve whose area gives probabilities, and the Laplace density falls off exponentially in the size of each unit. Its log gives the absolute value penalty:

Its negative log is the penalty plus a constant that does not depend on the code. The constant can be dropped during learning, so the sparsity penalty behaves like a log prior. The book makes a careful remark here. Weight decay has a Bayesian reading as a prior over the parameters. This penalty depends on the data, so it is not a prior in the formal sense, but it still expresses a preference over functions. The book's view is that training the sparse autoencoder approximates maximum likelihood training of a generative model, so the penalty is a consequence of the model's distribution over its latent variables.

In the demo below, the Laplace density is the prior on one code unit. The strength sets how sharp the peak is, and the threshold marks a band around zero. Move both and see how much probability the prior places inside the band. A sharp peak puts most of the probability close to zero, and the kink of the absolute value at zero is what pushes codes to exactly zero.

The book describes one way to get exact zeros in the code: use rectified linear units (units that output their input when it is positive and zero otherwise) for the code layer. With a prior that pushes the code toward zero, the book notes, one can indirectly control the average number of zeros in the representation.

Denoising Autoencoders

The second route does not add a penalty. It changes the input. A denoising autoencoder (DAE) is trained on the corrupted input and must reconstruct the clean input . Compare the usual objective with the denoising one:

In (14.8), is a loss such as the L2 norm, the length, of the difference between the input and its reconstruction. The ordinary objective (14.8) can be met by copying, so the machine is free to pass every detail through. The denoising objective (14.9) cannot be met that way. The input has lost information about , so the machine must undo the corruption. Undoing it requires the structure of the data, which copying does not need.

The demo below uses a simple corruption: Gaussian noise (random values drawn from a bell curve) of spread , which is its standard deviation, added to each coordinate. The clean points lie on a line, and the denoiser is the best linear one that the 30 clean points allow. It keeps the part of the noisy point along the line, shrinks that part by a factor set by the noise, and drops the part across the line. Its exact form is added here, and is not printed in the book.

The denoising criterion has a further property the book stresses. A denoising autoencoder learns the structure of the data distribution as a side effect of learning to undo corruption, so the learned features are useful beyond the copy task. The book also notes that overcomplete, high-capacity models can serve as denoising autoencoders, as long as care is taken to prevent them from learning the identity function. Section 14.5 returns to this with the vector field that the denoiser learns.

Regularizing by Penalizing Derivatives

The third route penalizes the slope of the code. The penalty is added to the reconstruction loss, as in the sparse case, but it is the squared size of the gradient of each code unit with respect to the input:

The penalty acts only at the training examples. It asks the code to change little when the input changes a little around each training point, so the code captures the variation that the training data shows. The autoencoder regularized this way is a contractive autoencoder, treated in section 14.7. The book notes theoretical links between this penalty and denoising autoencoders, manifold learning and probabilistic modeling.

Take one sigmoid unit , an S-shaped function of its input that always lies between 0 and 1, with weight and bias . Its slope with respect to the input is the one below. The penalty is times the square of this slope, averaged over the inputs in the demo. This formula is added here and is not printed in the book:

The book applies this penalty to the Jacobian of the encoder, the table of slopes of every code unit with respect to every input coordinate. The demo shows the mechanism for one unit. Where the sigmoid is steep, the slope is large and the penalty pays a price. A unit saturates when its output sits near 0 or 1 and hardly changes. There the slope is near zero and the penalty costs almost nothing.

• A sparsity penalty on the code (14.2) pushes most units toward zero, and with a Laplace prior it is a log prior (14.5 to 14.7).
• A denoising objective (14.9) replaces copying by restoration, so the machine must learn the structure of the data.
• A derivative penalty (14.10 and 14.11) keeps the code changing little when the input changes slightly, so the code responds mainly to variation that the data show.
• In each case the autoencoder can have a wide code and still learn something useful.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.2 (pages 504–508), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.