Section 1
Spreading one unit of probability
How to use this pageClick the colored symbols. Every symbol, in an equation or in the text, opens its meaning. Move the controls before you read the paragraph under each demo, and commit to a guess in every Predict first box before you look at the answer.
Colors: ink for states, counts and parameters, teal for random variables and probabilities, violet for samples and shares, gold for tiny lengths.
random variables, PMFs and densities states, counts and parameters samples and shares uniform density, samples and shares tiny lengths
The previous module said that a random variable lists the possible states and that a distribution says how likely each one is. A probability distribution is that description, for one variable or for several at once. How it is written depends on whether the variables are discrete or continuous.
Predict first. A die has face 6 at a chance of 0.2, and the other five faces share the rest. You raise the chance of face 6 to 0.5. What happens to the other five faces?
The other five keep their ratios and share what is left, which is 0.5. Their total falls from 0.8 to 0.5, so the six chances still add to 1. The demo below sets face 6 to 0.5 when you press the answer.
Raise one chance and the others follow
chance of the face (the distribution)share of rolls so far
A picture that works for both cases is a liter of water. Exactly one liter exists, and all of it must land somewhere. For a discrete variable you pour it into separate buckets, one per state, and the amount in each bucket is that state’s probability. For a continuous variable there are no buckets. The water forms a thin layer along a line, and the depth of the layer at a point is the density.
Samples follow the distribution
the distributionwhat the samples show
In the die tab, the sample bars match the six chances more closely as the sample grows. In the continuous tab, each bar’s height is its share divided by the bin width. That division turns a share into a depth of probability per unit length, which is what a density is. The die chances and the two bumps are invented toy data from a seeded generator, not values from the book.
Section 2
Probability mass functions
Predict first. Five states get the values 0.2, 0.3, 0.1, 0.25 and 0.2. Is this a probability mass function?
Each value is between 0 and 1, but the total is 1.05. A PMF needs values that add to exactly 1, and the first demo in this section uses the same five values.
A distribution over discrete variables is described by a probability mass function, or PMF, written with a capital . It maps each state of to the probability that takes that state. The book writes that probability as
Click any coloured symbol to see what it means.
and for short, . A value of 1 means the state is certain, and a value of 0 means it is impossible.
What makes a function a PMF
Three properties are required. The domain of must be the set of all possible states of . Every value lies between 0 and 1 inclusive, since no state is less probable than impossible or more probable than guaranteed:
Click any coloured symbol to see what it means.
Predict first. You push the value of state A below zero and leave the others alone. Which property of a PMF fails?
The value of A is now below 0, so the second property fails. The total may also change, but the rule that values stay between 0 and 1 breaks first. The demo below sets state A to a negative value when you press the answer.
Is it a PMF?
value of the functionviolates a rule
Push a value below zero and the second property fails. Push one above 1 and the same property fails from the other side. Leave every value legal but move the total away from 1, and the third property fails.
Third, the probabilities must add up to exactly 1, a property the book calls being normalized:
Click any coloured symbol to see what it means.
The two displays restate the book’s bullet points in symbols. They are not numbered in the book. Without normalization, adding up the probabilities of several outcomes could give more than 1.
Challenge +8 XPSet the five values by hand so that every value is between 0 and 1 and the total is exactly 1. Do not use the rescale button.
The rescale button divides each value by the total, which repairs a total that is wrong while every value is non-negative. It cannot repair a negative value, because dividing by a positive total leaves the sign unchanged.
When several variables are in play, the reader must infer which PMF is meant from the variable, not from the letter. and are usually different functions that share a name. The book also writes first and gives its distribution later with the symbol :
Click any coloured symbol to see what it means.
Joint distributions
Predict first. The grid has two weather rows and three commute columns, so six cells. Before any counts are entered, what must the six cell probabilities add up to?
Whatever the counts, the cells are shares of one grand total, so they add up to 1. The demo below sets all six counts to 10 when you press the answer, so each cell holds one sixth.
A joint distribution from counts
The counts are an invented survey of how people traveled on dry and wet days, for illustration only. Click a cell to read it as a sentence. Change any count and every cell changes, because each one is divided by the new grand total, but the grid always adds to 1.
A joint distribution is easiest to see as a grid, with one row for each state of and one column for each state of . Each cell holds one probability, and the whole grid adds to 1.
Challenge +8 XPSet the counts so that one cell holds exactly a quarter of all the people. The readout shows the share of each cell.
A PMF can act on many variables at once. A distribution over several variables is a joint probability distribution. The expression
Click any coloured symbol to see what it means.
is the probability that equals and equals at the same time. The book shortens it to . The rules do not change. The domain is now the set of all pairs of states, and the normalization runs over every pair:
Going deeper: from counts to a joint probability (added)
This step is added here and is not printed in the book. A survey that records people in all counts each combination once. The chance that a randomly chosen person falls in one cell is that cell’s count divided by the grand total :
Click any coloured symbol to see what it means.
Each count is at least zero, and the counts sum to the grand total, so the cells sum to 1. That is the normalization rule from section 2 applied to pairs.
Section 3.4 of the book reads single-variable distributions off a grid like this one.
Section 4
The uniform distribution over states
Predict first. Eight states are equally likely, and you draw 600 samples. About how many samples fall in each state?
Each state has probability 1/8, so the expected share is 600 divided by 8, which is 75. Your draws will scatter around 75. The demo below sets the number of states to 8 when you press the answer.
For a discrete variable with different states, the uniform distribution makes each state equally likely by setting
Click any coloured symbol to see what it means.
for every state . The value is positive because is a positive integer. It is also at most 1, because is at least 1. That last remark is added here. The sum over the states is
Click any coloured symbol to see what it means.
so the distribution is properly normalized. In words, slices of size make one whole.
Slicing the unit equally
probability 1 over share of draws
Slide to 1 and the single state is certain, with probability 1. Slide it up and each bar shrinks while the total stays at 1. The shares bounce around the expected level, and a long run settles near it.
Challenge +8 XPSet to 8 and keep drawing until you have at least 1200 samples and every state’s share is within 0.03 of 1/8. Each press of the draw button adds 600 samples.
Probability density functions
Predict first. A narrow, tall bump has a peak of about 2.7. You shade the interval from −0.5 to 0.5. About what probability does the shaded area hold?
The bump is so narrow that almost all of its area sits within half a unit of zero. The area is about 0.999, even though the peak is above 2. The demo below selects the narrow bump when you press the answer.
Probability is area
densityarea between a and b
Start with the narrow bump. Its peak is 2.66, and yet the shaded area between −0.5 and 0.5 is 0.999, because the curve is only tall where it is thin. Widen the interval and the area approaches 1 but never passes it. Narrow the interval and the area approaches 0, which agrees with the last module: a single exact point has probability zero.
Challenge +8 XPSelect the wide bump and shade an interval whose area, the probability, is between 0.49 and 0.51.
For continuous variables the tool is a probability density function, or PDF, written with a lower case . A function must have the right domain, which is the set of all possible states of . It must also be non-negative everywhere:
Click any coloured symbol to see what it means.
The book adds that there is no requirement that stay at or below 1. Finally, the total area under it must be 1:
Click any coloured symbol to see what it means.
Predict first. A density is required to be non-negative, with total area 1. Must it also stay at or below 1 everywhere?
The book does not require p(x) to be at most 1. A density is probability per unit of length, so a narrow bump can rise above 1 while its total area stays 1. The narrow bump in the demo above peaks at 2.66.
The displays restate the book’s bullet points. Why is there no upper limit of 1? The density at a point is not a probability. It is the depth of the water layer, which is probability per unit of length.
The book says that the probability of landing inside an infinitesimal region with volume is given by the depth times that volume:
Click any coloured symbol to see what it means.
The tiny-region approximation
exact arearectangle p(x) times length
With a short region the curve is nearly flat, so the rectangle matches the exact area. As the region grows, the rectangle becomes a worse match. The box below explains the limit, which is added here.
Going deeper: why the rectangle is exact in the limit (added)
This derivation is added here and is not printed in the book. Over a short region the density changes little, so its value at the left end is a good stand-in for the whole region. The exact area is an integral. Taylor’s expansion of the density around gives
Click any coloured symbol to see what it means.
The error term shrinks faster than the length itself. Relative to the area, the error therefore falls as the length shrinks, which is what the readout shows for short regions.
Probability comes from adding up the density over a set, which means integrating it. The book states that the probability that lies in a set is the integral of over . The simplest set is an interval from to :
Click any coloured symbol to see what it means.
Geometrically, the probability is the area under the curve between and . The first demo in this section lets you choose the shape and the interval.
Section 6
The uniform distribution on an interval
Predict first. A variable is uniform on the interval from 2 to 3, so every value in it is equally likely. What is the probability that it lands between 2.2 and 2.5?
The density inside the interval is 1, because its width is 1. The query interval has length 0.3, and a density of 1 over length 0.3 gives 0.3. The demo below sets the interval from 2 to 3 and the query from 2.2 to 2.5 when you press the answer.
The continuous counterpart of equal slices is the uniform distribution on an interval. The book writes it as , where and are the endpoints, with . The semicolon means parametrized by, so is the argument and , are parameters.
The book writes the function inside the interval and outside it. The display below restates that in cases and is not printed in the book in this form:
Click any coloured symbol to see what it means.
A rectangle of area one
density query interval, probability equals its overlap
Outside the interval the function is zero, so no probability sits there. Inside, it is the constant . That is non-negative everywhere, and it integrates to 1, because the rectangle it forms has width and height . The book states this, and the derivation below is added here.
Going deeper: the uniform density integrates to 1 (added)
This derivation is added here and is not printed in the book. Inside the interval the density is the constant , so the integral of the density is the area of a rectangle with width and height :
Click any coloured symbol to see what it means.
The density is zero outside the interval, so it adds no area there, and the total is 1.
The book denotes the distribution of a variable that follows it with the symbol :
Click any coloured symbol to see what it means.
Narrow the interval below width 1 and the rectangle rises above 1, while its area stays 1. The query probability is the overlap length divided by the width. Draw samples to see the share approach it.
Challenge +8 XPMove the query interval so that it holds exactly half of the probability.
What to take away
- A PMF maps each state of a discrete variable to a probability. It needs the right domain, values between 0 and 1, and a total of exactly 1.
- A joint distribution gives a probability to every combination of states, and the whole grid adds to 1.
- The uniform distribution on states gives each state , and the total is .
- A PDF is non-negative and integrates to 1, but its value at a point is not a probability and may exceed 1.
- Probability is area: the integral of the density over a region. A tiny region of length holds about .
- The uniform density on the interval from to is the constant inside the interval and zero outside.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.3 (pages 56–58), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.