Section 1
Corrupt, then reconstruct
A denoising autoencoder (DAE) receives a corrupted data point and must predict the original, uncorrupted point. Think of a photo restorer who is given a scratched print and must return the clean picture. The restorer learns the usual shapes of pictures, and that is what makes the restoration possible.
The book sets up the training in three steps. Sample a training example . Sample a corrupted version from the corruption process ( given ). Use the pair as one training example. The training criterion, which the book writes as an expectation over both samples, is equation (14.14):
Click any coloured symbol to see what it means.
The encoder receives the corrupted point , and the decoder must output the clean point . If the encoder is deterministic, the denoising autoencoder is an ordinary feedforward network, and it can be trained with the same methods.
Predict firstThe corruption is switched off, so the corrupted point equals the clean point. What does the autoencoder learn?
Corrupt the points, then see how far they moved
Challenge +8 XPSet the corruption so the mean squared corruption is between 0.2 and 0.3.
clean pointscorrupted points
The demo draws 30 clean points and one fixed set of Gaussian noise, scaled by the spread you choose. The mean squared corruption is the average of the squared distance each point moved. The demo uses a fixed seed, so the same noise is scaled as you move the slider.
Book 14.5.1
Learning a vector field
The score of a distribution is the gradient of its log density, a vector field that points toward higher probability. The book notes that learning this gradient field is one way to learn the structure of the data distribution itself. Score matching (Hyvärinen, 2005) is an alternative to maximum likelihood that encourages the model to have the same score as the data at every training point, and section 18.4 discusses it further:
Click any coloured symbol to see what it means.
A very important property of DAEs is that, with a conditionally Gaussian reconstruction distribution, the autoencoder learns a vector field (()) minus that estimates the score. The training criterion is the squared error (14.16), with the Gaussian corruption (14.17):
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Predict firstThe data sit at two peaks, at minus 1 and plus 1. What does the learned vector field do exactly at a peak?
For the demo, take a data density with two equal peaks. Its noisy version, after Gaussian corruption, is also a two-peak density, with wider peaks. For a Gaussian corruption, the best reconstruction has a known form. This identity is added here for the demo and is not printed in the book:
Click any coloured symbol to see what it means.
The vector field is the score of the noisy density scaled by squared. It is zero at the peaks, and it points toward the nearer peak elsewhere. The book's figure 14.4 draws the same idea around a curved manifold, where each arrow points toward the nearest point on the manifold. Its figure 14.5 shows the arrows shrinking where the data are dense.
The vector field of a two-peak density
Challenge +8 XPSet the spread so the vector at 0.5 points toward +1 with length at least 0.25.
Book 14.5.1.1
A short history
The book traces the ideas in order. Press each milestone in turn to read it. The next one unlocks after the one before.
Walk through the milestones
Challenge +8 XPRead all six milestones, in order.
The book also gives a remark on purpose. Like sparse autoencoders and contractive autoencoders, the denoising autoencoder was motivated by the need to learn a high-capacity encoder without the encoder and decoder learning a useless identity function.
Book 14.5.1
Fixed noise blurs the distribution
The book notes an important limit. With a fixed noise level, regularized score matching is not a consistent estimator. It recovers a blurred version of the distribution. Consistency is recovered only if the noise level goes to zero as the number of examples goes to infinity.
The demo below shows the blur. The two peaks of the data each have spread 0.3. Corruption with spread widens each peak to a spread equal to the square root of 0.09 plus squared. When that spread passes 1, the two peaks merge into one. The formula is added here for the demo and is not printed in the book:
Click any coloured symbol to see what it means.
Blur the two peaks
Challenge +8 XPBlur the density until the two bumps merge into one.
- A denoising autoencoder is trained to map a corrupted point back to the clean one, with the loss in equation (14.14).
- With Gaussian corruption, its reconstruction minus input is a vector field that estimates the score, as equations (14.15) to (14.17) describe.
- The history runs from MLPs for denoising in 1987 to a family of shallow models in 2015.
- A fixed noise level blurs the learned distribution, so the estimate is consistent only as the noise goes to zero.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 14.5 (pages 510–515), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.