Section 1
Writing totals in the margin
How to use this pageClick the colored symbols. Every symbol, in an equation or in the text, opens its meaning. Move the controls before you read under each demo, and commit to a guess in every Predict first box.
Colors: teal for probabilities and the totals in the margins, ink for variables, violet for one state of the variable we keep, gold for the states we sum or integrate away.
probabilities and densities variables one state of the variable we keep states we sum or integrate away
Press the buttons to total the rows of the grid, then its columns. Watch where each total lands.
Totals in the margin
row being addedcolumn being addedtotal written in the margin
You hold the joint distribution of and , and you want the distribution of alone. The distribution over the variables you keep is the marginal probability distribution. A solid lit from one side casts a flat shadow that drops its depth. A marginal is the shadow of a joint distribution.
The name comes from bookkeeping. Write the states of down the rows and the states of across the columns. The total of a row goes in the right-hand margin, and that total is the marginal probability of the row's state.
Predict first. The grid has three rows and three columns of joint probabilities, and all nine add to 1. If you total the three row totals and add them up, what do you get?
The row totals are complete shares of the whole, so they add to 1 whatever the grid. The demo above totals the three rows when you press the answer.
The grid is an invented survey of weather against commute. Each margin adds to 1 by itself, because each is a complete distribution over its own variable. The numbers are invented for illustration, not taken from the book.
Challenge +8 XPWrite all three row totals and all three column totals in the margins. The readout confirms that each margin adds to 1.00.
The sum rule
Take discrete random variables and with a known joint distribution . The marginal of comes from the sum rule, Equation 3.3 in the book:
Click any coloured symbol to see what it means.
Read it from the inside out. Fix one state and list every pair that contains it. The symbol adds their probabilities, which removes from the picture.
With the weather grid, the rainy row holds 0.04, 0.16 and 0.20, so the chance of rain is .
Predict first. The rainy row of the survey holds 0.04, 0.16 and 0.20. What is the probability that it rains, whatever the commute?
Adding the three cells of the rainy row gives 0.40. The demo below shows the same total as the top of the rainy bar when you press the answer.
Marginal as a stacked total
first state of the summed variablesecond statethird state
Each bar is one state of the variable you keep. Its pieces are the joint probabilities that the sum rule adds, and they are gold because they belong to the states we sum away. Click a bar to see its addition written out.
Predict first. Under the Unrelated table, what is the walk piece of the rainy bar?
The rainy row of the Unrelated table is 0.08, 0.12 and 0.20, so its walk piece is 0.08. The walk column total, 0.20, adds over the weather states instead.
Under Unrelated, every row splits across the commute states in the same proportions. Under Linked, the pieces of a bar can concentrate on one state, as the rainy bar does. Unrelated and Linked have the same row and column totals. The Survey table has the same row totals, but its column totals are 0.26, 0.37 and 0.37.
Going deeper: why the sum rule holds (added)
This derivation is added here and is not printed in the book. The event is the union of the events over all states of . Those pairs cannot overlap, because takes exactly one value in each outcome. Probabilities of disjoint events add, which gives
Click any coloured symbol to see what it means.
The sum rule is this line read from right to left. Nothing is missed, since every outcome with = falls in exactly one of the pairs.
Challenge +8 XPSwitch the model to Linked, then click the rainy bar. Its total is the chance of rain, 0.40.
What the margin discards
The sum rule throws information away. Different joint distributions can share the same margins, as different solids can cast the same shadow. This section is added here and is not in the book.
The demo has two yes-or-no variables, and . The first two sliders set the margins. The third moves probability between the four cells and keeps every row and column total unchanged.
Predict first. You slide the tie from its middle position to its strongest value. Do the two margins change?
The tie only rearranges probability inside the grid, so every row and column total is fixed. The demo below moves the tie to its strongest value when you press the answer.
Same margins, different joints
joint probability in a cell, darker is larger; margins sit beside the grid
Predict first. Set the tie to its middle position, where it is zero. With margins 0.6 and 0.3, what is the yes and yes cell?
At the middle position each cell is the product of its margins, so the cell is 0.6 times 0.3, which is 0.18. The demo above sets the tie to zero when you press the answer.
At the middle position each cell equals the product of its two margins. That is the case where knowing one variable says nothing about the other, which Section 3.7 of the book calls independence. At the two ends, the variables are tied as strongly as the margins allow, in opposite directions. A model that stored only the margins could not tell these joints apart.
Challenge +8 XPSet the tie so that the yes and yes cell reads between 0.04 and 0.06, while the two margins stay at 0.6 and 0.3.
Integrating out a continuous variable
For continuous variables a sum cannot list the states, because there are too many. The book replaces the sum with an integral, Equation 3.4:
Click any coloured symbol to see what it means.
Here is the joint density. For one fixed , the curve of is a slice through the joint density. The integral is the area under that slice, and that area is the marginal density at the position. The idea matches the discrete case, with area in place of a total.
Predict first. You change the tie between the variables from 0.6 to -0.9. Does the marginal density of change?
The integral over y adds up every slice at each x, and the tie only changes how those slices are shaped. The demo below sets the tie to -0.9 when you press the answer, and the marginal in the middle does not move.
Slices of a density add up to the marginal
joint density (darker is higher)marginal density of xslice at x, whose area is the marginal there
The joint density here is a toy mixture of two correlated bumps, generated for this demo. The numbers are invented and are not from the book. Slide the slice left and right. The shaded area under the slice in the bottom chart is exactly the height of the marginal curve at that position.
Predict first. The slice sits at . What is the marginal density there?
The marginal curve is about 0.13 at x = 0. It only reaches 0.30 near x = -1.2. The demo above moves the slice to zero when you press the answer.
Change the tie and the heat map leans one way or the other, while the marginal of stays where it was. The tie only rearranges probability along for each , and the integral over removes that rearrangement.
Going deeper: why integrating the joint density gives the marginal (added)
This derivation is added here and is not printed in the book. Take any region of the x axis. The probability that x lies in does not depend on y, so it equals the probability that x lies in the region and y is anywhere. Writing both probabilities as integrals of the joint density gives
Click any coloured symbol to see what it means.
The equation holds for every region , so the functions inside the integrals over must agree. The bracket is therefore the marginal density, which is Equation 3.4. The integral over y removes the tie between the variables, because it adds up every slice of the joint density at that .
The marginal of is found the same way, by integrating over .
Challenge +8 XPMove the slice to a position where the marginal density is at least 0.3. The readout gives the marginal at the slice.
Summing out several variables
The book speaks of the marginal over a subset of the variables. The subset can hold one variable or several, and any number can be summed away. With three variables, the marginal of alone adds over every pair of states of the other two:
Click any coloured symbol to see what it means.
This extension is added here. It repeats Equation 3.3 once for each variable removed, and the order of the sums does not matter.
Predict first. All three variables, , and , are kept, and each is yes or no. How many bars does the chart show?
Three yes-or-no variables have 2 times 2 times 2 joint cells. Keeping all three leaves each cell as its own bar. The demo below keeps all three when you press the answer.
Choose which variables to keep
Click a cell to take 0.05 from it, then click a second cell to receive it. The joint distribution changes with each move, so the bars change too. The cells are invented.
Keep all three variables and each bar is one joint cell. Drop one and neighboring cells merge. Drop all three and one bar of height 1 remains. The bars always add to 1, because summing a normalized distribution over some variables leaves a normalized distribution over the rest.
Challenge +8 XPKeep only , then move probability until the bar for reads 0.60.
Challenge +8 XPFind a set of kept variables whose bars include one that reads 0.10 and one that reads 0.40.
Predict first. You move 0.05 from one cell to another. Do the bars still add to 1?
The move takes 0.05 out of one cell and puts it into another, so the sum of all cells is unchanged. Every set of bars then adds to 1 as before.
What to take away
- A marginal distribution is the distribution over a subset of the variables of a joint distribution.
- For discrete variables, the sum rule adds the joint probabilities over every state of the variable removed (Equation 3.3). The name comes from writing the totals in the margin of a grid.
- For continuous variables, the sum becomes an integral over the removed variable (Equation 3.4).
- Margins are complete distributions on their own, so each adds to 1.
- Different joint distributions can share the same margins, so the marginals lose information about how the variables relate.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.4 (pages 58–59), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.