Applications of Autoencoders

Reducing dimension, the first use

The book names two applications of autoencoders: dimensionality reduction and information retrieval. This section takes the first. Dimensionality reduction maps each input to a shorter code , and the number of numbers in the code is its code size . It was one of the early motivations for studying autoencoders, and one of the first uses of representation learning and deep learning.

The book gives one famous example. Hinton and Salakhutdinov (2006) trained a stack of restricted Boltzmann machines (RBMs), used their weights to start a deep autoencoder with smaller and smaller layers, and ended at a bottleneck of 30 units. The book reports that this code had less reconstruction error than principal component analysis (PCA) into 30 dimensions, and that its categories formed well-separated clusters. This page does not repeat that experiment. Its demos use PCA alone, which is the linear baseline, on toy data, so the labels say exactly what is shown.

The toy data are generated, not taken from a real problem. Eighty points come from four clusters in a two-number hidden space. A fixed random matrix maps each hidden point to eight numbers, and every number gets noise of size 0.3. The data are then centred, so their average is zero.

PCA keeps the directions with the largest spread. Sort the directions by the spread they carry. Call the spreads , with the largest. A code of size keeps the first directions and drops the rest. The reconstruction error per point is then the spread that was dropped:

This identity is added here and is not printed in the book. It holds for the PCA projection with the data centred. A code of size equal to the input size drops nothing, so the error is zero.

The data were built from two hidden numbers, so two directions carry almost all of the spread. The third to eighth directions carry noise. This matches the book’s high-level picture, but the toy is far smaller than the book’s experiment.

Why smaller codes help

The book gives three reasons to want a lower-dimensional code. Lower-dimensional representations can improve performance on tasks such as classification. Smaller spaces consume less memory and runtime. And many kinds of dimensionality reduction place semantically related examples near each other, which the book says helps generalization.

The toy can test the third reason. For each point, look at its nearest neighbour in the code, the other point at the smallest distance, and ask whether that neighbour belongs to the same cluster. The share of points that pass is the neighbour agreement. This measure is a choice made for this page, not a result from the book.

With one number, every point sits on one line, and a few points from different clusters land next to each other on it, as the bars show. With two numbers, the clusters separate. Past two, the extra numbers carry little spread in this toy, so the agreement does not change. The book’s claim about semantic neighbours is a general pattern. The toy shows it for one case only.

Retrieval with binary codes

The book treats information retrieval as the task of finding database entries that resemble a query. It says this task gains even more than usual from dimensionality reduction. In a low-dimensional space, search can be very efficient. If the code is also binary, every database entry can be stored in a hash table, a table keyed by the entry’s code. A query then returns every entry that shares its code.

The search can also reach slightly less similar entries. Flipping one bit of the query’s code gives a new key, and the entries under that key are close to the query. The book calls the whole approach semantic hashing and says it has been applied to text and to images.

The demo builds a six-bit code from six lines. Each line gives one bit: the side of the line on which the point lies. Crossing one line therefore flips exactly one bit. The lines are drawn at random with a fixed seed and are not learned. So the demo shows the lookup, not the learning. The book’s binary codes are learned.

The bucket of a query is the set of entries whose six bits match the query code . With radius the search returns only the bucket. With radius it also returns the buckets one bit away, and with radius the buckets two bits away. Drag the point on the plane, or use the sliders, and watch the counts change.

Bits from saturated sigmoids

To make the code binary, the book uses an encoding function with sigmoids on its final layer. A sigmoid output is saturated when it is close to 0 or close to 1, where the curve is flat. The book says the sigmoid units must be trained to be saturated for all inputs, so that each one gives a clear bit.

The book suggests one trick. Add noise just before the sigmoid during training, and let the magnitude of the noise grow over time. Noise can push a unit across the middle of the sigmoid and flip its bit. To fight the noise, the network must increase the magnitude of the sigmoid’s inputs until the units saturate. The book states this in one paragraph. The demo below makes it measurable.

A bit survives the noise when the noisy unit gives the same bit as the clean one. Let the clean pre-activation have spread , that is a standard normal number times , and let the noise have spread . Then the chance that a bit survives is (added here, not printed in the book):

The formula comes from the chance that two correlated Gaussian numbers have the same sign. It is exact for this model. When is large compared with , the arctan is small and nearly every bit survives. When the two spreads are equal, the share is 75%. The demo holds the noise fixed, so it does not show the book’s growing noise, only the effect of the scale.

This demo does not train a network. It evaluates the formula above, and it draws 16 sample units from one seeded draw, so the strip shows the chance as a sample. The percentage in the readout is the exact expectation.

The book also notes that the hashing function can be learned directly, with a loss tied to finding nearby examples in the hash table (Norouzi and Fleet, 2011). This page does not cover that work.

• Autoencoders reduce dimension, and a code that keeps the top directions of the data keeps most of its spread. Reconstruction error is the spread that was dropped.
• A smaller code can keep semantic neighbours together, which the book says helps classification and generalization.
• A binary, low-dimensional code lets a hash table answer a query with one lookup, and flipping bits widens the search.
• Sigmoids that saturate give bits that survive noise. The noise trick works because larger inputs keep units saturated.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.9 (pages 524–525), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.