Contractive Autoencoders

Two forces and the Jacobian penalty

The contractive autoencoder adds an explicit penalty on the code = . The book writes it as equation 14.18. The penalty weight multiplies the squared Frobenius norm of the Jacobian of the encoder, so the penalty is small when the encoder's derivatives are small:

The Jacobian is the matrix of those partial derivatives. Row i belongs to code unit i and column j to input coordinate j, so entry (i, j) says how fast code unit i changes when input coordinate j moves. The Frobenius norm is the square root of the sum of the squared entries, so the penalty is the sum of the squares of every partial derivative of the encoder.

The penalty works against the reconstruction error. Reconstruction error alone would make the autoencoder learn the identity function. The penalty alone would make the features constant in . The compromise is an encoder whose derivatives are mostly tiny, so that only a few directions in the input change the code strongly.

Move the entries and the penalty together. The Frobenius norm is never smaller than the largest entry. So an entry of 0.4 with a norm below 0.5 forces the other three entries to have squares that sum to less than 0.09. A small penalty therefore keeps every entry small, not only the largest one.

Contractive linear operators

The name contractive comes from linear algebra. A linear operator is contractive if the norm of stays at most 1 for every unit-norm . In other words, is contractive if it shrinks the unit circle, so that no direction is stretched beyond length 1. The book applies this idea to the Jacobian, which is a local linear approximation of the encoder.

A singular value is the factor by which a matrix stretches a unit vector along one of its two perpendicular principal directions. For a two-by-two matrix, the largest stretch of a unit vector is the largest singular value . The operator is contractive exactly when is at most 1. The closed form below is added here and is not printed in the book. It gives both singular values from the four entries:

Contractive does not mean small. The matrix diag(0.95, 0.9) has a Frobenius norm of about 1.31, and it is still contractive, because no direction is stretched beyond 1. Contractive is a property of the largest singular value, which is a different property from a small Frobenius norm. The Frobenius norm is never smaller than the largest singular value.

Tangent directions from singular vectors

Directions with a large change the code quickly. The book says these directions are likely to approximate the tangent planes of the manifold. The leading singular vectors, the directions with the largest singular values, are read as the tangent directions that the contractive autoencoder has learned. Some singular values stay above 1, because the reconstruction error keeps the directions with the most local variance.

Figure 14.10 in the book compares two estimates of tangent vectors at one dog image from CIFAR-10. One is local PCA, which works from nearby examples. The other is the leading singular vectors of the Jacobian of a contractive autoencoder. The book notes that the autoencoder forms more accurate estimates from limited data, because it shares parameters across locations that share active hidden units. That figure needs real images and a trained network. This page rebuilds the idea on a ring of points, and the ring toy is added here. It is not the book's experiment.

On the ring, the point at angle has a radial direction and a tangent direction. The toy encoder stretches the radial direction by and leaves the tangent direction unchanged, so its Jacobian has singular vectors along those two directions. Local PCA estimates the tangent from nine neighbours of the point, with a small seeded jitter. Set and move the point, then compare the two directions.

The leading singular vector is radial when the radial scale is above 1, and tangent when it is below 1. So the direction that the penalty keeps depends on how the encoder scales its directions, and not on the manifold alone. Reading the largest singular vectors as tangents therefore works only when the encoder stretches the tangent directions the most.

Saturating sigmoid units

The book notes that when the penalty is applied to sigmoidal units, one easy way to shrink the Jacobian is to make the units saturate. A unit saturates when its output sits at its limit, 0 or 1, where its curve is flat. Saturated units encode inputs with extreme values, which can be read as a binary code, and they spread the code over most of the hypercube, the box of all possible code values, that the units can span.

For one unit = sigmoid of , the derivative is . This formula is added here and is not printed in the book. It shows why saturation shrinks the Jacobian: the product is near zero when is near 0 or 1.

Saturation is a trade-off, not a free gain. A saturated unit barely responds to its input, so the code can lose the variations that separate nearby inputs. The penalty and the reconstruction error must balance, so the units saturate only where the data do not need a response.

Tied weights block the scale trick

The book warns that the penalty can be fooled. Let the encoder multiply its input by a small constant and let the decoder divide the code by . As approaches 0, the penalty approaches 0 without the encoder having learned anything about the distribution, while the decoder keeps reconstructing perfectly. Rifai et al. (2011a) prevent this by tying the weights of the encoder and the decoder, so that the decoder uses the transpose of the encoder's weight matrix.

The toy below uses one input and a scalar scale. With tied weights, the decoder multiplies the code by , so for an input of 1 the reconstruction is times the input. The loss is then plus . Without tying, the decoder divides by , and the reconstruction error is zero at every scale. This toy is added here and is not in the book:

Tying makes the best scale finite. With tied weights, setting the derivative of the loss to zero gives equal to , which is 0.75 for . Without tying, the loss keeps falling as goes to zero, so the encoder can satisfy the penalty by shrinking its scale. This is the reason the tied-weight rule matters for the contractive penalty.

Denoising and contraction

A denoising autoencoder is trained to reconstruct a clean input from a noisy copy of it. Alain and Bengio (2013) showed that, in the limit of small Gaussian input noise, the denoising reconstruction error is equivalent to a contractive penalty on the reconstruction function . Denoising autoencoders make the reconstruction function resist small but finite perturbations of the input. Contractive autoencoders make the feature extraction function resist infinitesimal perturbations, and they penalize the encoder rather than the full reconstruction.

The demo uses a toy reconstruction function . It is not the book's function. The exact expected miss is computed by averaging over Gaussian noise of spread . The penalty approximation keeps the expansion to second order in the noise spread: the squared reconstruction miss plus times the squared slope. Equation below is added here and is not printed in the book:

At an input where , the reconstruction term vanishes, and the approximation is the penalty term alone. Its relative gap then falls like the square of the noise spread, because the omitted terms are of higher order in the noise. Where differs from , the omitted term is of the same order in the noise as the slope term, so it is not negligible. At an input of 0.5 the gap is about 2 percent at a spread of 0.05 and about 21 percent at a spread of 0.2. The book's statement is a limit as the noise goes to zero, and the shrinking gap at the input 0 is that limit in action.

Deep contractive stacks

The penalty is cheap for one hidden layer, but it becomes much more expensive for deep autoencoders. Rifai et al. (2011a) trained a series of single-layer autoencoders, each one reconstructing the previous hidden layer. Each layer was trained to be locally contractive, and the composition of contractive layers is contractive too. The result differs from jointly training the whole deep model with a Jacobian penalty, but it keeps many of the desirable qualitative characteristics.

The bound behind this is a standard fact about products of matrices. It is added here and is not printed in the book. For two layers with Jacobians and :

The demo fixes two layers, each Jacobian normalized to largest singular value 1, and lets you multiply each layer by its scale. The stack's largest stretch is at most the product of the two scales, and it is usually smaller, because the two stretches rarely line up along the same direction. A first layer that expands can therefore still give a contractive stack, provided the second layer shrinks enough.

• The contractive penalty of equation 14.18 is the squared Frobenius norm of the encoder's Jacobian, weighted by lambda.
• A linear operator is contractive when its largest singular value is at most 1, which is not the same as a small Frobenius norm.
• The leading singular vectors point along the directions the encoder stretches most, which read as tangents only when the encoder stretches the tangent directions most.
• Saturating sigmoid units shrink the Jacobian, and tied weights stop the encoder from cheating the penalty by shrinking its scale.
• In the small-noise limit, the denoising reconstruction error becomes a contractive penalty, and a product of contractive layers stays contractive.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.7 (pages 521–523), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.