Representational Power, Layer Size and Depth

One hidden layer, many units

Autoencoders are often trained with a single layer for the encoder and a single layer for the decoder. That is not a requirement. Think of a tailor who fits a suit with straight cuts. Each cut is one hidden unit, and each extra cut lets the suit follow the figure more closely.

The universal approximator theorem says that a feedforward network with at least one hidden layer can approximate any function in a broad class, to any accuracy, provided it has enough hidden units. Here counts the hidden units of the one layer.

The demo below fits one hidden layer to a wiggly target curve. The points where each unit starts to bend are fixed and evenly spaced, shown as small ticks along the bottom. Only the output weights are fitted, by least squares. Each extra unit adds one more bend the curve can use, and the fit improves.

A shallow mapping from input to code can represent the identity along the data arbitrarily well. The book adds the limit: a single hidden layer cannot enforce arbitrary constraints on the code, such as sparsity. The demo shows accuracy, and the next sections show what depth adds.

Folding the input: depth and cost

The book makes two claims about depth. A deep encoder can approximate any mapping from input to code, given enough hidden units. And depth can exponentially reduce the computational cost of representing some functions, and the training data needed to learn them. The second claim is easier to see with a fold than with a curve.

The tent map folds the interval from 0 to 1 at its midpoint. Applying it again folds each half again. One fold uses two hidden units, which the formula below shows. The formula is added here for the demo and is not printed in the book:

Counting the pieces gives the second formula. Each fold doubles the pieces, while one hidden layer with units can have at most + 1 pieces. This formula is added here for the demo and is not printed in the book:

The book states the advantage without a construction. The fold is one simple case where the gap can be seen directly: with layers, the deep encoder uses times 2 units, and a single hidden layer needs minus 1 units for the same number of pieces.

Stacking shallow autoencoders

The book describes a common strategy for training a deep autoencoder: greedy pretraining, which trains a stack of shallow autoencoders first, one layer at a time. The book notes that we often meet shallow autoencoders even when the goal is a deep one.

The book also reports that deep autoencoders yield much better compression than corresponding shallow or linear autoencoders, citing Hinton and Salakhutdinov (2006). The stepper below follows the usual recipe for the stack. Stage 1 trains a shallow autoencoder on the data. Stage 2 freezes its encoder and trains a second shallow autoencoder on the codes the first one makes. Stage 2 cannot start before stage 1 is done.

The book describes the pretraining step, and the stepper follows the common practice of starting each stage from the code of the one before. The final fine-tuning of the whole stack is part of that practice rather than a claim printed on these pages.

Which claims the book supports

Four claims about depth are below. Mark each one true or false. The explanation appears after you choose, and it points to what the book says.

• One hidden layer with enough units represents the identity along the data, but its input-to-code mapping stays shallow.
• A deep encoder can approximate any input-to-code mapping, given enough hidden units, and depth can exponentially reduce the cost of some functions.
• Deep autoencoders can be trained greedily, one shallow autoencoder at a time, and they compress much better than shallow or linear ones in the experiments the book cites.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.3 (pages 508–509), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.