Directed Generative Nets

Sigmoid belief networks

A directed graphical model is a set of variables connected by arrows. Each variable is drawn from a distribution that depends only on its parents, the variables with arrows into it. To make a sample, start at the top and move down. This order is called ancestral sampling.

Think of a family of coins. Each child flips a coin whose bias depends on what its parents showed. The coins at the top have no parents, so their bias is fixed. Every coin below inherits its bias from the coins above it.

A sigmoid belief network uses this idea with binary units. Each unit is 0 or 1. Its probability of being 1 is the sigmoid of a weighted sum of its parents plus a bias. The sigmoid turns any real number into a number between 0 and 1.

In Equation 20.70, is the state of unit , holds the weights, and is its bias. The sum runs over the units with , so each unit depends only on the units before it. The weight is the coefficient on the state of parent . The sigmoid is a function, and it is written without a symbol.

Set the parents on or off, then move the weights and the bias. Each parent that is on adds its weight to the input, and the probability is the sigmoid of the total.

The book notes that sigmoid belief networks are universal approximators of distributions over binary variables, given enough depth, even when each layer is only as wide as the data (Sutskever and Hinton, 2008). Inference over the hidden units is intractable, so the book turns to learned inference networks and to the methods of section 20.9.

Differentiable generator networks

A generator network is a differentiable function that turns noise into data. The architecture fixes the family of distributions the generator can produce, and its parameters pick one member of that family. The simplest case is a single affine layer, Equation 20.71.

Here is a sample, is a mean, is standard normal noise, and is the Cholesky factor of . The Cholesky factor is a lower triangular matrix whose product with its own transpose gives the covariance. The next demo checks this with real numbers.

Nonlinear generators handle distributions that are hard to write down. The book's example is inverse transform sampling: draw uniformly between 0 and 1 and pass it through the inverse of the cumulative distribution function. That needs the inverse, which is often unavailable, so deep generators learn the function from data.

For an invertible, differentiable , the density of the output follows from the density of the noise. The Jacobian determinant measures how much stretches volume near . Equation 20.73 divides by its absolute value, so the total probability stays at one.

The toy generator is , where the slope is a positive number. Its inverse is , and its Jacobian is . This example is not printed in the book. Its density follows from Equation 20.73.

The bars are histograms of 4000 seeded samples. The black curve is the exact density from Equation 20.73, and the bars follow it. The readout splits the formula into its three parts at the marked point.

A generator can also emit the parameters of a conditional distribution. Equation 20.74 uses sigmoid outputs as the probabilities of Bernoulli variables, which are yes or no values with a chance of being 1, so the data are binary. Equation 20.75 averages over the noise to get the distribution of . For example, if the generator outputs 0.9 for half of the noise values and 0.1 for the other half, then the probability that the output is 1 is 0.5 times 0.9 plus 0.5 times 0.1, which is 0.5.

The book states the trade-off. A generator that emits conditional parameters can produce discrete data. A generator that emits samples directly can produce only continuous data, because rounding the samples would break back-propagation. Both forms are trained with the reparametrization trick of section 20.9.

The second demo uses the Cholesky factor. The noise is standard normal in two coordinates, and is . Its covariance is , so the samples form an ellipse whose shape the covariance sets.

The variational lower bound

A variational autoencoder, or VAE, has three parts: a prior over a code , a decoder that generates data from the code, and an encoder that maps a data point to a code. The encoder's output is an approximate posterior, a guess at which codes could have produced the point.

Think of a librarian who files each book under a few tags, and a reader who rebuilds the book from those tags. The tags are the code. The encoder files, the decoder rebuilds, and training makes the rebuilt book close to the original.

Training maximizes a variational lower bound, also called the evidence lower bound or ELBO, on . Equation 20.76 writes the bound as a joint log-likelihood plus an entropy , which measures how spread out the encoder is.

Equation 20.77 rewrites the same bound as a reconstruction term minus a Kullback-Leibler divergence . The first term rewards codes that rebuild the data. The second term pulls the encoder towards the prior. Equation 20.78 says that the bound never exceeds the true log-likelihood.

The demo uses a toy with a closed form. The prior is standard normal, the decoder is normal around with spread , and the encoder is normal with mean and spread . For these choices the true posterior is also normal, so the gap in Equation 20.78 can be computed exactly.

Move the encoder mean and spread, and watch the solid curve approach the dashed posterior. The gap is zero only when the two curves coincide. The gap also equals the Kullback-Leibler divergence from the encoder to the posterior, which the readout computes separately as a check.

Latent coordinates and importance weighting

When the code is two-dimensional, it can be drawn on a plane. Figure 20.6 of the book shows images generated from a grid of codes. For faces, one axis mostly rotates the head and the other changes the expression. The book uses the figure to show that a trained encoder and decoder can lay out the data on a predictable coordinate system.

Figure 20.6 has a second panel for handwritten digits. The toy above does not reproduce those images, and the book's figure is described here, not rebuilt. The lesson carries over: a two-dimensional code can hold two independent factors that the decoder maps to visible changes.

The book extends the variational lower bound. The importance weighted autoencoder averages several samples of the code, with weights . Equation 20.79 uses samples.

With the objective is the ordinary lower bound of Equation 20.76. With more samples it tightens, and it stays below , which is the sense in which it estimates the true log-likelihood with importance sampling. The demo uses the toy of section 3, with the decoder spread fixed at 1 and the datum fixed at 1.5.

The curve is estimated from 4000 seeded draws, so the value at can differ from the closed form of section 3 by a few thousandths.

Generative adversarial networks

A generative adversarial network, or GAN, pits two networks against each other. The generator makes samples. The discriminator outputs the probability that an input is real. Think of a counterfeiter who learns from a detective. Each improvement on one side forces an improvement on the other.

The game has a value , and the generator's payoff is minus that value. Equation 20.80 states the goal, and Equation 20.81 gives the default value.

The discriminator wants large, which means it classifies real data as real and generated data as fake. The generator wants small. At convergence the generator's samples match the data, and the discriminator outputs one half everywhere.

The book warns that simultaneous gradient steps on a game need not converge. Its example is a value , where one player controls and the other controls . The next demo uses this example to show what finite steps do.

The book's circular orbit appears in the limit of infinitesimally small steps. This analysis is not printed in the book. With a finite step, the update below multiplies the distance from the origin by at every step, so the path spirals outward.

The second demo of this section shows the best discriminator. For a fixed generator, the discriminator that maximizes Equation 20.81 at every point is given below. This formula is not printed in the book. Where the two densities are equal, it outputs one half, which is the book's convergence statement.

The book reports that the best formulation in practice is not zero-sum. The generator maximizes the log probability that the discriminator errs, which keeps its gradient large when the discriminator is confident. Training is still hard, and the book calls stabilization an open problem. Dropout in the discriminator matters, and the book reports that never using dropout seems to yield poor results.

Conditional GANs learn distributions of the form , where is a condition such as a class label. LAPGAN uses a series of conditional GANs to add detail to a low-resolution image. The book's Figure 20.7 shows bedrooms from DCGAN and churches from LAPGAN. Those photographs are not rebuilt here.

Moment matching and maximum mean discrepancy

A moment of a random vector is an expectation of a product of powers of its elements. The mean is the first moment, and the mean of the squared values is the second. For a vector of powers , a general moment has the form in Equation 20.82, where each entry of is a non-negative integer.

The moment matching idea trains a generator so that its moments agree with the data. Matching every pair already needs a number of constraints that grows with the square of the dimension, and even the first two moments of a Gaussian only capture linear structure. GANs avoid listing moments by letting a discriminator choose the statistic to match.

Generative moment matching networks use a maximum mean discrepancy, written . It compares the averages of many feature maps through a kernel . It is zero if and only if the two distributions are equal. The kernel used in the demo is a Gaussian with width , and the estimate below is not printed in the book.

The estimate uses batches. It cannot be computed from one sample at a time, because the moments are averages over many samples. The book notes that small batches can underestimate the variation in the distributions, and that larger batches are slower per step.

Convolutional generators and un-pooling

A recognition network shrinks the image as it climbs: pooling discards detail and keeps what the label needs. A generator reverses the flow. It adds detail as it goes, so the spatial size must grow. The book uses the transpose of the convolution described in section 9.5 for this, which often gives more realistic images with fewer parameters than fully connected layers.

Pooling cannot simply be inverted, because most pooling functions throw information away. The book's workaround is un-pooling. It places each value at the upper-left corner of a block of size and fills the other cells with zeros. This is the exact inverse of max-pooling only under strong assumptions: the stride equals the block width, the maximum sits in the upper-left corner, and every other input in the block is zero.

The restored grid keeps only the maxima that sat in the upper-left corner. The other cells are zeros. The layers after un-pooling are trained to fill in the gaps, which is why the samples of the whole model look reasonable even though the operator itself is unrealistic.

The chain rule and linear auto-regressive networks

This section covers two book subsections, 20.10.7 and 20.10.8. An auto-regressive network has no latent variables. It models the joint distribution of the observed variables with the chain rule of probability, which the book writes in words. The chain rule is shown below as a display:

Each conditional may depend on all earlier variables. Such models are called fully visible Bayes networks, or FVBNs. Figure 20.8 of the book draws the graph: each variable has arrows from every variable before it.

The simplest case gives each conditional a linear model, which is logistic regression for binary variables. There are no hidden units and no shared parameters. Each conditional has one weight per earlier variable and one bias.

For binary variables, the conditional of variable has a table of free entries, so the full model needs numbers. The linear model needs numbers, which the book summarizes as order . These counts are not printed in the book.

The book notes that linear auto-regressive networks inherit the limits of linear classifiers. They can be trained with convex losses and sometimes have closed forms, as in the Gaussian case. Their capacity cannot grow by adding units, so the book raises it with basis expansions or the kernel trick.

Neural auto-regressive networks and NADE

This section covers two book subsections, 20.10.9 and 20.10.10. A neural auto-regressive network keeps the same left-to-right graph but replaces each conditional with a neural network. A table would need one entry for each configuration of the earlier variables. A network needs far fewer parameters and can still capture high-order dependencies. Figure 20.9 shows the hidden units arranged in groups, where group depends only on the first inputs. Those groups can be reused for every later prediction.

NADE, the neural autoregressive density estimator, adds one more form of parameter sharing. The weights from input to the element of every group with are the same. Equation 20.83 writes this rule, and Figure 20.10 draws it with matching line styles.

The ratio of unshared to shared weights is , which the schematic makes visible. The arrows of one colour carry one weight column. The count is not printed in the book.

NADE can be extended to mimic several steps of mean field inference, and the book calls that NADE-k. For continuous data, the conditionals can be a Gaussian mixture. Equation 20.84 averages models over many orderings of the variables, which often generalizes better. Its ensemble is built from the same kind of conditionals.

Each ordering gives a complete distribution, and an average of distributions is again a distribution. That is why the eight ensemble values sum to one, whatever the number of orderings.

For continuous values, the book uses the RNADE model. Its conditional is a mixture of Gaussians with mixture weights , means and variances . The weights come from a softmax, so they are positive and sum to one. The variances are made positive. The ensemble of Equation 20.84 is defined for any model that gives a probability for each ordering. The mixture density is written below for reference, and it is not printed in the book.

• A sigmoid belief network draws each unit from a sigmoid of its parents, and sampling runs from the top down (Equation 20.70).
• A generator maps noise to data through a differentiable function. Its density follows from the Jacobian (Equations 20.71 to 20.73).
• A VAE maximizes a lower bound whose gap is the Kullback-Leibler divergence from the encoder to the posterior (Equations 20.76 to 20.78). More samples tighten the bound (Equation 20.79).
• A GAN is a two-player game (Equations 20.80 and 20.81). Finite steps on that game can spiral outward, and a generator that matches the data makes the best discriminator equal one half.
• Moment matching compares averages of kernel feature maps, and the maximum mean discrepancy is zero only when the distributions are equal (Equation 20.82 defines the moments, and the discrepancy estimate is not printed in the book).
• Auto-regressive models factor the joint distribution with the chain rule. NADE shares its weights (Equation 20.83), and averaging over orderings gives an ensemble (Equation 20.84).

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.10 (pages 692–710), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.