Undercomplete Autoencoders

A copy machine with a bottleneck

Imagine describing a photograph to a friend on a postcard with room for only two numbers. The friend must redraw the photo from those two numbers alone. You would soon learn which features matter most, because the rest cannot fit on the card. An autoencoder plays this game. It has an encoder that maps the input to a code , and a decoder that maps the code back to a reconstruction . The chapter opens with this picture.

The training goal sounds silly at first. We want the output to copy the input, so that undoes exactly:

Copying alone teaches nothing. If the code is as wide as the input, the machine can pass every detail through. So the trick is to make the copy hard. Put a bottleneck in the middle: a code narrower than the input. An autoencoder whose code is narrower than the input is called undercomplete. Below, the data are two-dimensional, so a code with one number is undercomplete and a code with two numbers is not.

The one-number code in the demo uses the best straight line, found by the method of section 3. That is a fair test of what a linear bottleneck can keep.

The reconstruction loss

The autoencoder learns by making the reconstruction close to the input. The book writes the training objective as a loss , a number that is small when the reconstruction is good and large when it is poor. The loss depends on the input and on the output of the whole machine:

In this equation is the loss, and is the input. The inner part (()) is the reconstruction. The book gives the mean squared error as one example of . It squares each coordinate of the difference between the reconstruction and the input, then averages the squares. Before averaging, the sum of the squared coordinate differences is the square of the straight-line distance between and its reconstruction.

The shape of the loss is worth seeing once. Fix one input and move the reconstruction along a line. The loss is zero exactly at = , and it grows like a parabola away from it.

An undercomplete autoencoder has a code with numbers, where is smaller than the input size . A complete code has equal to , and an overcomplete code has more numbers than the input. Section 14.2 treats the last two cases. Here we keep the code narrow.

Read the formula above as the loss averaged over training inputs, with the reconstruction through a one-direction code. The sum over is the average over the training examples. This formula is added here for section 3 and is not printed in the book.

Linear decoders and PCA

The book states one precise fact. If the decoder is linear and the loss is mean squared error, an undercomplete autoencoder learns to span the same subspace as principal component analysis (PCA). PCA is the method that finds the directions in which the data vary most. A subspace is a flat piece of space through the origin, such as a line or a plane. For a code with one number, the subspace is a line through the origin.

Let the code direction be the unit vector (), with angle . The encoder takes the dot product with , which multiplies matching coordinates and adds the results. The decoder multiplies the number back by . The reconstruction is the projection of the point onto the line, which is the closest point on the line:

The error then depends only on the angle . Turn the line until the squared distances to the points are as small as possible. Equivalently, turn it until the points spread as far as possible along it. The demo does exactly this. The curve plots the error for every angle, and the lowest point is the best line. The data are centred, so their mean is zero, which is the usual setting for PCA.

The best angle has a closed form. Write , and for the sums of squares and products of the centred coordinates. The lowest error is then at the angle below. This formula is added here for the demo and is not printed in the book:

The book's claim and this formula agree: the best line is the first principal direction. The error at that line is the variance left over across the line, the average squared spread of the points in the second direction. So the code keeps exactly the direction that carries the most variance, and that is why the undercomplete code learns the principal subspace.

Be careful with the scope of the claim. The book says the learned subspace is the same, which for a one-number code means the same line. Nonlinear encoders and decoders are not covered by this result, and section 4 explains why.

Too much capacity: the memorizer

A nonlinear undercomplete autoencoder can learn a more powerful version of PCA, a curved summary of the data rather than a flat one. The book describes an extreme case. Take a code with one number and a very powerful nonlinear encoder. The encoder can give each training example its own integer code , and the decoder can learn to map each integer back to its example. The copy is then perfect, and the machine has learned nothing about the data's distribution. The book adds that this scenario does not occur in practice, but it shows clearly how a copy task can fail when capacity is too great.

Think of a library catalogue that stores each book under a number. The number lets you fetch the book exactly, but it says nothing about what the book is about. The demo below builds this memorizer on six training points in the interval from 0 to 1.

The lesson is one the book states directly. A high-capacity autoencoder can copy its training data without learning useful properties, so the design must limit what the code may keep. Section 14.2 takes up this problem and gives the autoencoder other properties to prefer, without shrinking the code.

• An autoencoder maps an input to a code and back, and is trained to make the output match the input.
• An undercomplete code has fewer numbers than the input, so the machine must keep only the structure that matters.
• With a linear decoder and squared error, the best one-number code is the first principal direction of the data (PCA).
• With enough capacity an autoencoder can memorize its training inputs and learn nothing, which is why section 14.2 adds other preferences.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.1 (pages 503–504), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.