Common Probability Distributions

Named families of distributions

Any model of a quantity has to say how likely each value is. A real number has too many values to list, so statisticians use a family of distributions: one formula with a few adjustable numbers, the parameters. Each choice of parameters picks one member. A coin needs one parameter, the chance of heads. A bell curve needs two: its center and its width .

On the bell tab the Gaussian wins, because its two numbers are all the data need. On the two-humps tab no choice of two numbers gives two humps, while a histogram with enough bins finds both. Push the bins up with few training points, and the histogram starts fitting single points, so its score collapses.

Bernoulli distribution

The simplest random variable has two outcomes: a coin lands heads or tails, a message is spam or not. Call the outcomes 1 and 0. The Bernoulli distribution has one parameter, , in [0, 1], which is the probability that . The book gives the first two properties as Equations 3.16 and 3.17:

The two outcomes exhaust all possibilities, so the second follows from the first. One formula covers both values, Equation 3.18:

It is a switch written as arithmetic. For the exponent of is 1 and the exponent of is 0, so the formula returns . For it returns . Because one formula covers both cases, it can be multiplied over many observations and differentiated. The mean and variance are Equations 3.19 and 3.20:

With few tosses the observed share jumps around, and with thousands it settles on . The variance is largest at one half, where the outcome is most uncertain, and it is zero at both ends, where the outcome is certain. A Bernoulli variable has no free spread: once is set, the variance is fixed.

Multinoulli distribution

A variable with different states, such as the next word in a sentence or the class of an image, needs the multinoulli or categorical distribution. It is parametrized by a vector of probabilities, where is the probability of the -th state. The last state gets whatever is left, and the entries must satisfy . In symbols:

Only numbers are needed, because the probabilities add up to 1. For this is exactly the Bernoulli distribution.

For three states, plot against the second probability. Non-negativity and a total of at most 1 select a triangle. Each point inside is a valid distribution, and the third probability is the distance to the sloping edge.

The corners are the three certain distributions, where one state has probability 1. The center is the uniform distribution. The count vector after draws follows the multinomial distribution, and the readout evaluates it for the counts you see.

Gaussian distribution

The most used distribution over real numbers is the normal or Gaussian distribution, the bell curve (Equation 3.21). Its plot is the book's Figure 3.1.

The factor in front is a normalizing constant that makes the total area 1. The mean is any real number, and the standard deviation is positive. The variance is .

The book marks the peak at and the inflection points at . An inflection point is where the curve stops bending down and starts bending up. Drag the round handle to move and the square handles to change .

A smaller makes the peak taller and the bell narrower. The area stays 1 because the peak height is . The share of probability within one, two and three standard deviations depends only on the distance in units of , so it is the same for every and .

The precision parametrization

Evaluating the density needs inverted and squared each time. The book therefore gives Equation 3.22 with the precision , the inverse of the variance:

The distribution does not change. High precision means a narrow bell, and a precise measurement is one with small variance. The gain is computational: when the density is evaluated many times, no division by is needed.

Reasons for the Gaussian default

Without prior knowledge of the shape of a distribution over the real numbers, the normal is a good default. The book gives two reasons.

The central limit theorem

Many quantities are sums of many small influences: measurement noise, sensor error, many small random effects. The central limit theorem says that the sum of many independent random variables is approximately normal, whatever the shapes of the individual terms, as long as no single term dominates. A complicated system can therefore often be modeled as normal noise even when its parts are not normal.

With one term the histogram is the source itself. On the default trials, the uniform and two-spike sources are already close to a bell at 5 terms, with a distance of about 0.01 to 0.05. The exponential source needs about 20 to 30 terms to reach about 0.03. The biased coin is slowest, because a sum of coin tosses takes only whole-number values: its histogram is a comb of spikes that follows the bell, and even at 60 terms the distance is about 0.1.

Maximum uncertainty at fixed variance

The second reason is that, among all distributions with the same variance, the normal encodes the maximum uncertainty over the real numbers. It inserts the least prior knowledge into a model. The book defers the full argument to Section 19.4.2. To compare densities we need a measure of uncertainty, the differential entropy , which Section 3.13 defines as the average surprise of a draw. It is not printed in this section:

The Gaussian bar sits at zero and every other density lies below it. A change in moves all the entropies together, so the ranking never changes. The two-bump mixture shows the effect best: at zero separation it is a Gaussian, and as the bumps move apart at fixed variance, its uncertainty falls. This is a handful of examples, not a proof.

Multivariate Gaussian

The normal generalizes to , where is the dimension. It has a mean vector and a positive definite symmetric covariance matrix (Equation 3.23):

Compare it with one dimension. The squared distance has become the quadratic form , the squared Mahalanobis distance.

The matrix is the covariance matrix from Section 3.8. Its diagonal holds the variance of each coordinate and its off-diagonal entries the covariances. The density needs the inverse, which is expensive to compute again and again, so Equation 3.24 uses the precision matrix , the inverse of the covariance:

A diagonal covariance makes the coordinates uncorrelated, and for a Gaussian uncorrelated means independent. The isotropic Gaussian goes further: its covariance is a scalar times the identity matrix, so the variance is the same in every direction and the contours are circles.

Exponential and Laplace distributions

A Gaussian is smooth everywhere, so it cannot place a sharp point at one value. The exponential distribution places its peak exactly at zero (Equation 3.25):

The symbol is the indicator function. It equals 1 when its condition holds and 0 otherwise, so every negative value gets probability zero. For the density starts at height and decays at rate .

A closely related distribution places a sharp peak at any point : the Laplace distribution (Equation 3.26).

It is two exponential tails glued back to back at . The decay uses the absolute distance where the Gaussian uses the squared distance. This gives a pointed peak and heavier tails at the same scale. The mean is and the variance is (standard, not printed).

Dirac and empirical distributions

Sometimes all the probability should sit at one point. The tool is the Dirac delta (Equation 3.27):

The delta is zero everywhere except at 0, yet it integrates to 1, which no ordinary function can do. The book calls it a generalized function, defined by what it does inside an integral. One can think of it as the limit of functions that put less and less mass away from 0. A shift by gives an infinitely narrow, infinitely high peak at . Its key property is that integrating it against a function picks out that function's value at the spike.

As the width shrinks the bell grows taller and thinner, its area stays 1, and the weighted average of slides toward . In the limit this is the sifting property at work.

The empirical distribution

A common use of the delta is the empirical distribution of a data set (Equation 3.28):

With data points, each spike has mass . The spikes add up to 1, so this is a valid distribution. It is the distribution we sample from when we train on the data.

To sample from it means picking a data point at random, with replacement. A model trained on the data is therefore asked to match this spiky distribution. How spiky should the model be? The next demo replaces each spike by a bump of width .

As the spikes narrow, the likelihood of the training data keeps rising, because each point sits on top of its own tall spike. In the limit of zero width, the training likelihood grows without bound. The score on new data tells the other half of the story. It peaks at an intermediate width and then falls steeply. The empirical distribution reproduces its training set and says almost nothing about new points. The two goals, fitting the data and generalizing from it, pull in different directions.

Mixtures of distributions

A mixture distribution combines component distributions. On each trial, a component identity is drawn from a multinoulli distribution, and the sample is then drawn from that component (Equation 3.29):

Here is the multinoulli distribution over the identities. In words: the probability of is the sum over components of the chance of choosing that component times the chance that it generates . The empirical distribution is a mixture too, with one Dirac component for each data point.

Latent variables

The identity is a latent variable: a random variable that we cannot observe directly. It is linked to the visible through the joint distribution. Latent variables return in Section 16.5.

With the color box ticked, each sample is generated in two steps: a draw of the hidden component and then a draw from it. Untick the box and only the dots remain, which is all an observer sees. The prior is the model's belief about before it sees , and the posterior is the belief after. The book makes this distinction with the words “prior” and “posterior”.

Gaussian mixtures

A very powerful and common type is the Gaussian mixture model, whose components are Gaussians with their own means and covariances . The covariances can be shared, or each component can be restricted to be diagonal or isotropic. Besides these, the parameters include the prior probabilities of the components. Figure 3.2 shows samples from a three-component model whose components illustrate the three kinds of covariance. The next demo rebuilds it.

Uncoloured, the samples look like a cloud with three lumps, which is what real data look like. Colored, the mixture shows how it generated them. The isotropic lump is round. The diagonal lump is stretched along one axis. The full lump is stretched along a direction that only a full covariance matrix can express.

Universal approximation of densities

The book's last claim here is that a Gaussian mixture is a universal approximator of densities: any smooth density can be approximated, with any specific non-zero error, by a Gaussian mixture with enough components. A single component is limited, but adding components removes the limit. The demo fits a mixture with components to a target density, by expectation-maximisation run on the target itself. It starts from equally spaced components and adjusts every weight, mean and width. The error is the Kullback-Leibler divergence , which is 0 only when the two agree.

With one component the error is largest for the two pointed peaks. The flat top and the skewed target are next, and the triangle is already fairly bell-shaped. The error then falls as components are added, though not smoothly, because expectation-maximisation can settle in a local optimum. For the two pointed peaks the error is 0.0026 nats at six components, 0.0010 at twelve and 0.0009 at sixteen, so it levels off. The guarantee is for smooth densities, and sharp features are the hard case.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.9 (pages 62–68), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.