Technical Details of Continuous Variables

Densities, sets and integration

The book builds probability for continuous variables on one rule from Section 3.3.2. The probability that a continuous vector lands in a set is the integral of its probability density function, written , over that set.

Two facts follow at once. A density is not a probability: it is a probability per unit length, so it can be larger than 1. As the set shrinks to a point, its probability shrinks to zero, while the density there does not.

Set the spread to 0.2 and the peak density rises to about 2, a perfectly legal value, because the bell is narrow and its area is still 1. Now drag the width to 0. The probability of an interval around the peak falls to exactly 0 although the curve is still at its highest there. In a continuous world, any single outcome has probability zero, and positive probability belongs to stretches of the line. Tick the box and 50 extra isolated points join the set. The probability does not move, since each point adds zero.

Sets that cannot be integrated over

The book flags a warning. Some choices of produce paradoxes. One can build two sets and with no point in common for which . Probabilities of disjoint sets can add to at most 1, so these sets break the basic rules.

The book keeps two other pieces of measure theory, because they state theorems about most points while allowing a few exceptions. They are the next two sections.

Measure zero

Measure theory gives a way to say that a set of points is negligibly small. Such a set is said to have measure zero. The book does not define it formally and gives the intuition: a set of measure zero occupies no volume in the space where it lives. Length, area and volume are the three everyday cases. The book’s examples are:

• In the plane, a line has measure zero, because it has no area. A filled polygon has positive measure.
• An individual point has measure zero.

Thickening a set

Here is a hands-on way to see why a line is negligible and a polygon is not. Thicken the shape by a distance on every side, and ask what fraction of the square its thickened version covers. The demo scatters 20000 random points in the unit square and counts how many fall within of the shape.

Drag towards 0.001 for the point and the line. The covered fraction collapses: for the line it is about proportional to , for the point to . Halving halves the line’s fraction and quarters the point’s.

Then switch to the triangle. Its fraction stops shrinking at the triangle’s own area, because a filled shape has an inside that thickness cannot take away. That inside is what positive measure means.

Countably many points still add up to nothing

Countably many means the points can be listed one after another. A union of countably many sets of measure zero also has measure zero, so the rationals, which can be listed this way, are negligible. This is the surprising case, since they are packed densely into every stretch of the line.

List the rationals in as a first, a second, and so on. Pick any tiny number . Cover the first with an interval of length , the second with one of length , and the -th with length . Every rational is covered, and the intervals together have total length at most

The demo shows the first rationals from a list ordered by denominator.

Since can be as small as you like, the rationals fit inside a set of total length as small as you like, so their measure is zero. The argument is added here and is not printed in the book.

The tick marks are the rationals. The violet bars are the intervals. They shrink so fast that after a handful of them they are thinner than a pixel, and the total length climbs towards but never passes it. That is the whole reason a countable set can be ignored.

Almost everywhere

The second term the book borrows from measure theory is almost everywhere. A property holds almost everywhere if it holds throughout the whole space except on a set of measure zero. Since the exceptions occupy a negligible amount of space, they can be ignored for many purposes.

The function used as a neuron activation has a kink at 0 and no derivative there, but it is differentiable everywhere else. This example is added here and is not drawn from the book.

With one kink the function is exactly the ramp . Each added kink makes the slope jump at a new place. The readout counts how many of the 5000 random samples hit a kink exactly, and the answer is 0 every time. Samples do come close: the fraction within of a kink shrinks in proportion to , which you can watch by moving the slider.

The exceptions exist, they form a countable set, and a continuous random draw never finds them. Away from the kinks the slope agrees with a numerical central difference to many digits, which is also reported.

This is the sense in which results hold “almost everywhere”: statements about nearly every point, with a negligible exception list.

Transforming a random variable

The third technical detail concerns variables that are deterministic functions of one another. Take , with invertible, continuous and differentiable. If has density , it is tempting to plug the inverse into :

The book says that this is not the case, and that it is a common mistake. Its example is with , the uniform distribution on the interval from 0 to 1, whose density is 1 on that interval and 0 outside. For this map the inverse is , so the naive rule reads:

This equals 1 on the interval from 0 to one half and 0 elsewhere. The book then writes

and a probability density must integrate to 1. The error is that the naive rule ignores how distorts space. A small region of length around carries probability , and if stretches or squeezes that region, the density must change by the opposite factor.

The corrected rule

The book asks for the two tiny regions to carry equal probability:

Write . Divide out the -region length for the density of , and the -region length for :

The only new piece beyond the naive rule is the stretch factor, shown in orange. It is the absolute value of the derivative, because lengths are positive. Where squeezes space the density of rises, and where stretches space it falls.

For the factor is , so on the interval from 0 to one half, which integrates to 1. The next demo compares the naive and corrected rules for four transformations against histograms of samples.

In every tab the histogram follows the teal curve and not the orange one. In the first tab the naive curve has the right shape but half the height, so its area is 1/2, the book’s Equation 3.43.

The areas shown in the box are computed by integrating over the values of , which is the naive density after a change of integration variable. They are exact up to numerical error, and they equal the areas of the orange curves.

In the second tab the naive rule integrates to 1 and still fails to match the histogram: the true density is large near 0, where squaring squeezes the interval from 0 to 0.1 down to the interval from 0 to 0.01. Normalization would not have caught this error, which is a reason to prefer the corrected rule over checking the area. For the log-normal tab and the tanh tab the naive curve has the wrong area as well.

The Jacobian in higher dimensions

With vectors, the stretch factor becomes a matrix. The book calls it the Jacobian matrix, the matrix of first partial derivatives of the transformation. For real-valued vectors the book’s result is Equation 3.47:

The matrix inside is , the derivatives of itself.

The picture matches one dimension, with areas in place of lengths. The densities differ by the ratio of cell areas, which is the absolute value of the determinant. Pick a cell on the left to see its image on the right.

In the linear tab the cell and its image always have the same area ratio, , wherever you place the cell, because a linear map stretches space the same way everywhere. Make the determinant small and the parallelogram thins out, and the density on it rises to so that the total stays 1.

Try the matrix that sets and so that the determinant is 0: the square collapses onto a line, its area is 0, and no density in the plane can describe the result. This is why the map must be invertible.

The polar tab is the case where the stretch factor changes from place to place. Points are drawn uniformly in and converted with . A cell at radius is stretched by exactly , so near the center the same probability is squeezed into very little area: the dots crowd together there.

The density in the plane is , which grows without limit as . A uniform distribution in one coordinate system is not uniform in another, and the Jacobian is the bookkeeping that tells you how.

What to take away

• Probabilities come from integrating a density over a set. A density is a probability per unit volume, so it can exceed 1, and a single point has probability 0.
• Pathological sets can break the rules, and measure theory says which sets are safe. Everyday sets such as intervals and boxes are safe.
• A set of measure zero occupies no volume. Points, lines in the plane and countable unions of them have measure zero, and so do the rational numbers.
• A property that holds almost everywhere fails only on a set of measure zero, which a continuous random draw never reaches.
• If with invertible and differentiable, then . Plugging the inverse in without the stretch factor is wrong.
• For vectors the stretch factor is the absolute value of the Jacobian determinant, and it can vary from place to place.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.12 (pages 71–73), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.