Section 1
Independence as a factorization
probability, joint or marginal
conditional probability
size of a mismatch
, variables and values
How to use this pageClick any colored symbol. Inside an equation, its meaning appears under the equation. In running text, a small note appears. Answer each prediction before the demo reveals the result, and move the sliders before you read the sentence under them.
Read the symbols first. A random variable is written in an upright letter, such as , and one of its values in an italic letter, . The symbol means “for all”, so the equation must hold for every pair of values. Two variables are independent when the joint probability of every pair equals the product of the two marginal probabilities:
Click any coloured symbol to see what it means.
On the left is the joint probability of the pair. On the right are the two marginals, multiplied. If one single cell fails, the variables are dependent.
Predict first. The demo starts with and . Set the dependence knob to 0. What is the joint probability of ?
Test a joint table for independence
Two yes/no variables. The three sliders choose the joint table: the probabilities that and are 1, and a “dependence” knob that moves probability between the diagonal and the off-diagonal cells without changing either marginal. The picture has one column for each value of , 0 on the left and 1 on the right, as wide as its probability. Inside a column the teal part is and the violet part is , as tall as . The dashed line marks where the split would sit if ignored . The knob changes no marginal, which makes a point worth remembering: the marginals alone cannot tell you whether two variables are independent.
split if ignored
Challenge +8 XPMove the dependence knob until the worst mismatch in the table is above 0.15. The mismatch is the largest gap between a joint cell and its product of marginals.
Put the dependence knob at 0. Every cell matches its product and the dashed line runs exactly along the border between teal and violet in both columns. Move the knob away from 0 and the border stops lining up with the dashed line.
A useful consequence: when the variables are independent, the conditional probability ignores the condition. Dividing the joint by as in Equation 3.5 gives, whenever :
So under independence the probability of is the same whatever turned out to be. (This consequence is added here and is not printed in the book.)
Section 2
Independence in finite data
Equation 3.7 speaks about a probability distribution, which is an idealized object. In practice you meet a data set, and its table of frequencies is only an estimate of the distribution. Even if two variables are truly independent, a finite sample will almost never satisfy Equation 3.7 exactly, because counts fluctuate. So independence in data is a question of how big the mismatch is compared with what chance alone would produce. (This section is added here to connect the definition with data. The book defines independence only for distributions.)
Predict first. Two yes/no variables are truly independent. You draw samples. What is the typical size of the largest mismatch between a joint frequency and the product of the marginal frequencies?
The demo draws pairs from a table with . The strength knob sets the true dependence in the same way as before: the diagonal cells get 0.25 + and the others 0.25 − . At each sample size the plot shows the worst mismatch between the observed joint frequency and the product of the observed marginals.
How far from independent does chance look?
observed mismatchtrue dependencetypical chance mismatch, 0.20/√
Challenge +8 XPMake the true dependence at least 0.05 with a sample of at least 10,000, and let the observed mismatch rise above 0.02. The dependence should then be visible above chance.
With the strength at 0 the variables are truly independent, yet the violet curve is not zero. At ten samples it is large, and on average it falls like 1/√N as the sample grows. A single data set wiggles around that trend, and the curve touches the floor of the plot whenever the mismatch happens to be almost exactly zero.
Predict first. The strength is 0 and the data set has ten observations. Is the violet curve exactly zero?
The orange line is the typical chance mismatch, about 0.20/√N. (This size is derived here, not in the book: it is the mean absolute mismatch, which the curve plots.) Now raise the strength to 0.05. For small N the violet curve cannot be told apart from the independent case. For large N it flattens at the true value, because a real dependence does not average away.
In practice, a small mismatch in a small sample is not evidence of dependence, and independence can only be made plausible by data, never proved.
Going deeper: where the chance level comes from (added here, not printed in the book)
Take one cell, the pair x = 1, y = 1. Its mismatch is the observed frequency of that pair minus the product of the two observed marginal frequencies. Write each observation as a pair of 0/1 values. Up to a term of order 1/N, that mismatch equals the average of the centered product (xi − ½)(yi − ½) over the sample. Under independence with fair marginals, each term has mean 0 and variance (½)2 × (½)2 = 1/16. Averages of N independent terms have standard deviation sqrt(1/16 / N) = 0.25/√N. That is the standard deviation of the signed mismatch, the value the simulation confirms. The plot shows an absolute value, and the mean of |mismatch| under independence is sqrt(2/π) times that standard deviation, about 0.20/√N. The orange line uses this mean, so it is the typical size of the curve. The argument is leading-order: the leftover term is of order 1/N and disappears faster.
Conditional independence
Often two variables are not independent overall, but become independent once a third variable is known. The book defines this in Equation 3.8. The variables and are conditionally independent given if their conditional distribution factorizes for every value of :
Click any coloured symbol to see what it means.
This is Equation 3.7 applied separately inside each slice where is fixed. The probabilities are now conditional, and the test must pass for every , every and every .
Predict first. Ice cream sales and drownings rise and fall together across all days. On days with the same temperature, does ice cream sales still predict drownings?
Conditional independence and independence are separate properties, and neither one implies the other. In the common cause structure, and are both built from , so they are conditionally independent given but marginally dependent. In the collider structure, and are independent coin flips and is 1 when at least one of them is 1, so they are independent but conditionally dependent. (Both structures are added here and are not printed in the book.)
Predict first. In the collider, and are independent coin flips and exactly when at least one of them is 1. You learn and then . What is ?
Condition on z and watch the dependence move
The three panels show the joint distribution of and : overall, then with the condition set to 0, then with it set to 1. In the collider the panel for has only one possible outcome, so it is aligned automatically.
Challenge +8 XPCut the link: make the overall mismatch below 0.000001. Set equal to , or set the two probabilities for to be equal.
Challenge +8 XPSwitch to the collider tab and make the mismatch given above 0.2. Make both probabilities small, about 0.1 or less.
split if ignored
In the common cause structure, the two panels for the condition are aligned exactly when does not depend on within each slice. The overall panel is aligned only if or ignores , meaning its conditional probability is the same for both values of . Set and to the same value to cut the link, and the overall panel aligns too. Conditioning removes the dependence in the common cause, and in the collider it creates one.
Going deeper: why the collider creates a dependence (added here, not printed in the book)
Let x and y be independent with P(x = 1) = P(y = 1) = p, and let z = 1 when x or y is 1. Given z = 1, the cell x = 0, y = 0 has probability zero, because that combination makes z = 0. Its marginals, P(x = 0 | z = 1) and P(y = 0 | z = 1), are both positive, so their product is not zero. That single cell already breaks Equation 3.7 inside the slice z = 1. The size of the mismatch depends on p: it is about 0.14 at p = 0.4, about 0.22 at p = 0.1, and about 0.11 at p = 0.5. Both probabilities must be small, below about 0.19, for the mismatch to exceed 0.2. The collider challenge therefore needs both of them small.
Notation and why it pays off
The book introduces a compact notation for both properties. Independence is written with the perpendicular sign, placed between the two variables:
Click any coloured symbol to see what it means.
and conditional independence adds the conditioning variable after a bar:
Click any coloured symbol to see what it means.
The first statement means Equation 3.7 holds, and the second means Equation 3.8 holds. Both describe a distribution, not a data set.
The real value of these assumptions is economy. In the last module you counted the numbers needed for binary variables with the chain rule: the unrestricted chain needs . If all variables are independent, numbers suffice.
Three ways to describe n binary variables
full table, conditionally independent given , fully independent,
Challenge +8 XPMake the full table larger than a billion numbers while the middle road stays below 100. Find the band of where both hold.
A middle road is common in practice. Suppose all variables are conditionally independent given one hidden class variable . Then you need one number for , and for each variable one number for each of the two values of . That is numbers in total. (The class variable and the counts are added here, derived from Equations 3.7 and 3.8 with the chain rule. They are not printed in the book.)
Predict first. Ten binary variables are conditionally independent given one binary variable . How many free numbers does the model need?
With 40 variables the full table needs more than a trillion numbers, while the other two models need 40 and 81. The price is that the model makes a claim about the world that may be false. If the variables in your problem really influence each other directly, forcing independence throws that information away. Choosing which independences to assume is the central modeling decision behind the structured probabilistic models of chapter 16.
What to take away
- Variables are independent when the joint distribution equals the product of the marginals for every pair of values (Equation 3.7). One failing cell is enough to break it.
- Equivalently, under independence the conditional distribution of does not change with .
- Finite data never matches the product exactly. A mismatch of order 1/√N is what chance alone produces.
- Conditional independence (Equation 3.8) is independence inside each slice where a third variable is fixed. It neither implies nor follows from independence.
- Independence assumptions turn exponentially many parameters into a manageable number, at the cost of a claim about the world.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.7 (pages 60), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.