Section 1
A joint distribution built from questions
probability, joint or marginal
conditional probability
surprise in bits
, variables
How to use this pageClick any colored symbol. Inside an equation, its meaning appears under the equation. In running text, a small note appears. Answer each prediction before the demo reveals the result, and move the sliders before you read the sentence under them.
A joint distribution gives a probability to every combination of values of several random variables at once. For three yes/no variables, , and , the joint table has one entry for each combination of values.
Predict first. Each variable can be 0 or 1. How many entries does the joint table have?
For a hundred variables the table has about 1030 entries, so nobody writes it down. The chain rule builds the table from much smaller pieces.
Think of a game of twenty questions. To describe one possible day you ask: Does it rain? Then, knowing the answer: Is the road jammed? Then, knowing both: Do you arrive late? Each question has its own small table, and the table you consult depends on the answers so far. The probability of the whole day is the product of the probabilities along the way.
Predict first. The demo below starts with and . Before you look, how probable is together?
The book states the tool needed to make this precise in Section 3.5. The conditional probability that given that is
Click any coloured symbol to see what it means.
Read it as a ratio of two areas. The numerator is the probability that both hold. The denominator is the probability of the known value, so the fraction gives the share of the cases where that also have . It is defined only when . Multiplying by the denominator gives:
Click any coloured symbol to see what it means.
In words: two things together have the probability of the first times the probability of the second given the first. The chain rule repeats this.
Build a joint distribution from three small tables
Toy story, invented for this lesson: = 1 means it rains, = 1 means the road is jammed, and = 1 means you arrive late. The rows of the diagram read rain, jam and late. Each cell is labeled with its value. Teal means 1 (yes), violet means 0 (no). Try this: set the two conditionals for jam to the same value, then raise the probability of the late cell under rain and jam to 0.98, and watch the bottom cell for all three 1s widen.
Click a cell in the bottom row to follow its path.
Challenge +8 XPMake the outcome cover at least 90% of the strip. Raise all three probabilities that lead to it.
Each row of the diagram is one question. The top row splits the strip by , the second row splits each piece by the rule for given , and the third row splits again. A cell is as wide as the product of the numbers on its path, which is its probability.
Two things follow. The eight bottom cells always fill the strip exactly, and seven numbers, 1 + 2 + 4, define all eight probabilities. Setting the two conditionals for equal makes irrelevant to . That is independence, the topic of Section 3.7.
Section 2
The rule and its derivation
The diagram generalizes to any number of variables. Write them as up to , the superscript in brackets numbering them and not being a power. Equation 3.6 in the book says that any joint distribution is one marginal followed by a chain of conditionals:
Click any coloured symbol to see what it means.
Read the right side from left to right. The first factor, , is the first variable alone. Each later factor conditions one more variable on all the ones before it. The product sign means multiply these together. The book calls this the chain rule and says it follows from Equation 3.5.
Predict first. The table gives and . What is ?
The book prints the three-variable case in three lines. Apply the product rule once, conditioning on and together:
Click any coloured symbol to see what it means.
The leftover factor is itself a joint distribution. Apply the rule to it, conditioning on :
Click any coloured symbol to see what it means.
Put the second line into the first. The chain rule for three variables then reads, as in the book's third line:
Click any coloured symbol to see what it means.
Check the derivation on numbers
The table of eight joint probabilities is generated from a seeded random function. It is a toy and means nothing. Choose one outcome and move the step slider to watch the derivation reproduce its table entry.
In step 1 the marginal is a sum over , the sum rule from Section 3.4. In step 4 the product equals the table entry to every digit shown. That is an identity, not an approximation: each division in steps 2 and 3 is undone by a multiplication in step 4.
For variables the same step repeats times. Each time, split off the last variable and condition it on all the others. Stop when only is left. (The general induction is added here. The book prints only the three-variable case.)
Going deeper: why induction covers any number of variables (added here, not printed in the book)
Call the statement for n variables S(n). For n = 1 it says , which is true. Now assume S(n − 1) and write the joint over n variables as the joint over the first n − 1 times the probability of the last one given all the others. That is the product rule of Equation 3.5 with as the event and the rest as the condition. Replace the joint over the first n − 1 variables by the chain of S(n − 1). The result is the chain for n variables, so S(n) holds. Equation 3.6 for every n follows.
Any order gives the same answer
The book numbers the variables 1 to , but the numbering is arbitrary. For three variables there are 3! = 6 orders, each a valid chain rule. The factors change with the order, but their product does not. (This remark is added here and is not printed in the book.)
Instead of multiplying small probabilities, look at the surprise of each factor, , measured in bits. A probability of 1/2 has a surprise of 1 bit, 1/8 has 3 bits, and a certain event has 0. The logarithm turns products into sums, so the outcome's surprise is the sum of its factors' surprises.
Predict first. The surprise of an event with probability 1/8 is how many bits?
Six orders, one total
A random toy table with its own seed, the same kind as before. Each bar is one order, numbered 1 to 6 in the order of the tabs. Inside each bar, factor 1 is teal and factors 2 and 3 are violet.
Every bar has the same total length, so only the division changes. The tab shows the factors of its order as numbers. If comes first, its factor is the plain marginal , and the factor for at the end conditions on both others.
The freedom is used in practice. Text models read words left to right, and image models may scan rows from top to bottom. The mathematics allows any order, and which one is easiest to learn is a modeling decision. (The examples of text and image models are added here and are not printed in the book.)
Section 4
Sampling and counting parameters
The rule as a sampling recipe
The chain rule also tells you how to sample from a joint distribution, which means drawing outcomes with the right frequencies. Draw from its marginal. Then draw each next variable from its conditional given the values already drawn, until . This is ancestral sampling, and it is how a language model writes text, one word at a time. (Sampling is not discussed on these book pages; it is added here as an immediate use of the rule.)
Predict first. The first bit is 1 with probability 0.7, and each later bit repeats the one before it with probability 0.8. Which sequence is more likely, 1111 or 0000?
Draw sequences one bit at a time
A toy model of four bits. The first bit is 1 with probability . Every later bit repeats its predecessor with probability “stay” and flips otherwise. The exact probability of a sequence is computed with the chain rule. The samples are drawn by the recipe above with a seeded random generator.
observed frequencyexact chain rule probability
Challenge +8 XPSet the stay probability to at least 0.90 and draw at least 10,000 samples. The two constant sequences, 0000 and 1111, must then hold more than half of the samples.
With ten samples the bars are ragged, and with ten thousand they sit on the markers. That is why the recipe is correct: frequencies converge to the products of conditionals. Raising the stay probability makes 0000 and 1111 dominate.
What the rule does not do
The chain rule does not, by itself, make large joint distributions cheap. The full table of binary variables needs numbers. The factor for needs one number for each of the combinations of the earlier variables, and gives the same count. The rule rearranges the information without compressing it. (This counting argument is added here.)
Predict first. Ten binary variables are modeled by a chain in which each variable sees only the one before it, so the memory is . How many free numbers does the chain need?
Savings appear only with an assumption. If each variable depends on at most earlier ones and is unaffected by the rest, the factor for needs numbers, and the total is
The minimum picks the smaller of and , so the exponent never exceeds the memory.
How many numbers does a model need?
full joint table, or the unrestricted chain rulechain with memory
Challenge +8 XPChoose at least 20 variables and a memory of at most 3. The chain must then need less than one thousandth of the full table.
The orange curve grows by a factor of two with every added variable. The teal curve grows only linearly once exceeds , because each new variable adds a fixed number of new entries. The assumption that a variable does not depend on the earlier ones is called conditional independence, which Section 3.7 defines. The chain rule is the exact bookkeeping, and conditional independence makes that bookkeeping affordable.
Going deeper: the closed form for a limited memory (added here, not printed in the book)
Split the sum at the memory. For the first terms the exponent is , and these form a geometric series that sums to . Every later term is , and there are of them. So the total is
With and this gives 3 + 8 · 2 = 19, which is the number in the prediction above.
What to take away
- The product rule is Equation 3.5 with the denominator moved across.
- Applying it repeatedly gives Equation 3.6: any joint distribution is the product of one marginal and a sequence of conditionals, each variable conditioned on all those before it.
- The order of the variables is free. Factors change with the order, the product does not.
- The rule is a recipe for sampling, one variable at a time.
- On its own the rule needs as many numbers as the full table. Savings come from independence assumptions, which Section 3.7 introduces.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.6 (pages 59–60), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.