Denoising Score Matching

Fitting the training points exactly

Suppose a model is flexible enough to match any density, and you train it by maximum likelihood on a few examples. Each example pulls the density up at its own location, so the best possible model piles its probability onto the examples and leaves almost nothing elsewhere.

Think of a spotlight that can be focused to a point. The spot is bright, and everything outside it is dark. The empirical distribution of a training set is the extreme form of that spotlight: a spike at each training point and no mass anywhere else. A model that copies it fits the training set perfectly, and fails on every new example.

The model needs a way to spread its mass out a little. Denoising score matching does this by fitting a blurred version of the data, and the blur is controlled by the smoothing width , the standard deviation of the noise added to each point.

The demo draws the density that each choice of produces. The six training points are fixed at −1.6, −1.1, −0.4, 0.3, 0.9 and 1.5. The model density at is the average of six bumps, one centred on each point, and each bump has standard deviation . The readout shows the average log-likelihood of the six points under that density.

The answer to the prediction is goes up. When is small, each point sits under its own bump of height close to , and the logarithm of that height grows without limit as shrinks. The readout checks this with the formula −log(6σ√2π), which becomes accurate once the bumps stop overlapping. Large spreads each bump across the whole axis, and the training likelihood falls.

Smoothing the data with a corruption process

Denoising score matching fits a smoothed density, defined by the book in Equation 18.27. The smoothed density is the data density averaged over a corruption process . That is the probability of getting the corrupted point from the clean point . The usual choice adds a small amount of noise to :

Read the integral from the inside out. For each clean point , the corruption spreads probability around it. The integral then adds up these spread-out clouds, each weighted by how common its clean point is in the data. The result is a blurred version of . The book notes that denoising score matching is especially useful because in practice we usually have only samples from the data, not the density itself, and it cites Kingma and LeCun (2010) for regularised score matching with normal noise as the smoothing distribution.

The book states that smoothing costs the consistency property of section 5.4.5. A model that fits cannot recover exactly for any above zero, however much data it sees. The price is paid to avoid the spikes of Section 0. The next demo measures that price on a density you can compare with the truth.

The corruption can be written in a form that is easy to use. Take a Gaussian corruption with noise scale , and replace by the empirical distribution of training points. Both steps are added here and are not printed in the book:

The dashed curve in the first panel is the true density, a mixture of two bumps at −1 and 1. The solid curve is the smoothed density built from the six training points. The lower panel scores each smoothing width by the average log-likelihood of new points drawn from the true density. This is the test log-likelihood. Its value is computed by summing over a fine grid, so no random draws are involved.

Two regimes show up. For a small width the curve is spiky and fits the six training points, but it assigns very low density to much of the true distribution, so the test log-likelihood is very negative. For a large width the curve is smooth but too flat to describe two separate bumps. The test log-likelihood peaks in between, near = 0.33 for this data. The best width depends on the data, which is why the choice of is a hyperparameter, a setting that you choose rather than one that training learns.

The denoising target

The score of a density is the gradient of its log with respect to the point . It is a vector of the same shape as , and it points in the direction in which the log-density grows fastest. Think of it as the slope of a hill under your feet: it tells you which way is uphill and how steep it is.

The denoising method needs the score of the corruption, not of the data. Take the Gaussian corruption from Section 1 and compute its log with respect to the point . The steps are added here and are not printed in the book:

The minus sign in the first step comes from the square in the exponent. The second form says that the denoising target points from the corrupted point back to the clean point, and that its length is the distance divided by squared. A corrupted point far from its source gets a long arrow, and a small noise scale makes every arrow long. The target needs only the pair (, ), so it can be computed from data without knowing .

Drag the corrupted point along the axis. The orange arrow is the denoising target, and its readout is the exact derivative of the log-density at that point. The second line of the readout computes the same slope by a small finite step. Both numbers agree, which is a check on the algebra and not a new result.

Move to the lowest and highest values. At the lowest, a point only a little away from the clean point already gets a long arrow, because the corruption is so narrow that any displacement is unlikely. At the highest, a point one unit away gets an arrow of length 1.00, because the corruption is so wide that a displacement of one unit is hardly surprising.

Denoising score matching on a Gaussian model

Denoising score matching trains a model score , a function that proposes an arrow at every point. The model is trained to match the denoising target from Section 2, so the loss compares the two arrows. This objective is added here and is not printed in the book:

Now take the smallest case that shows what the loss learns. The data are Gaussian with mean and spread , and the model is a Gaussian with mean and spread . The corrupted point is = + , with a standard normal draw. The model score is then minus , and the target is minus .

Expanding the square and taking expectations gives the loss in closed form. This derivation is added here and is not printed in the book. The corrupted point has mean and total spread , and the cross term contributes to the loss, so

Two facts follow from this formula, and the demo below lets you check both. For any , the smallest loss over is at , the data mean. Then, with the mean at the data mean, the smallest loss over is at . Together these give the joint minimum at and . The learned width is the total spread , which is wider than the data spread by the amount of the noise. The model has learned the smoothed density, not the data density.

The upper panel holds the spread fixed and moves the mean. The lower panel holds the mean fixed and moves the spread. The dashed line marks the minimum in each panel. The readout gives the smallest possible loss over both sliders, which the closed form predicts as , and the setting that reaches it.

The mismatch between and is the cost of smoothing in the Gaussian case. With a short noise scale the mismatch is small, and with a long one the model is noticeably too wide. Section 1 showed the same effect in the test likelihood.

Denoisers and the score

Section 14.5.1 of the book shows that several autoencoder training algorithms are equivalent to score matching or denoising score matching. A denoiser is a function that takes a corrupted point and returns its best guess of the clean point. The best guess under squared error is the conditional mean . The identity below links that guess to the score. It is added here and is not printed in the book, and the Gaussian case gives a direct way to check it:

The right side says that the reconstruction moves the input by times the score of the smoothed density. A denoiser is therefore a score estimate in disguise: the difference between its output and its input is the score multiplied by the noise variance .

For Gaussian data the denoiser is a straight line, given by the second equation below. This closed form is added here and is not printed in the book. Its slope , the gain, is the share of the total spread that comes from the data. The demo uses data centred at zero, so = 0 in the readout.

Drag the input point along the axis. The violet segment is the move the denoiser makes. The two numbers in the readout, the move and times the score, are equal at every input. The readout computes the score from the formula for the smoothed density, not from the denoiser, so the equality is a real check of the identity.

Setting the gain to one half needs the data spread and the noise scale to be equal. With no noise the gain is one, and the denoiser returns its input unchanged. With very large noise the gain is close to zero, and the denoiser ignores its input and returns the mean. This is the same trade-off as in Section 1, seen from the denoiser side.

• Fitting a density to a finite sample needs smoothing, because the exact fit is a set of spikes at the training points (Section 0).
• Equation 18.27 defines the smoothed density as the data density averaged over a corruption process. Smoothing gives up consistency, so the fitted density is not the data density.
• For Gaussian corruption the denoising target is , the noise scale squared divided into the gap between the clean and corrupted points. It is the score of the corruption, and it needs only that pair of points.
• On a Gaussian toy the denoising loss has its minimum at the data mean, and the fitted spread is the total spread, which is wider than the data by the noise.
• A denoiser moves its input by the noise scale squared times the score, which is the link to the autoencoder equivalence the book cites from section 14.5.1.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 18.5 (pages 619–620), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.