Section 1
Reversing a conditional
How to use this pageClick any colored symbol, in an equation or in a sentence, and it explains itself. Then make the guesses and move the sliders before you read on.
the posterior, the answer you want the prior, belief before the observation the cause, what you want to infer the observation, what you can see the likelihood the evidence, a normalizer
A conditional probability such as is the probability of the observation when you already know the cause . The order matters: and are different numbers.
One thousand people, one test
sick, positivehealthy, positiverings: test negative (sick = teal, healthy = gray)
The filled dots are the people who test positive. Teal ones are sick, and orange ones are healthy people the test got wrong. With the default numbers the test finds 90% of the sick, yet most positives are orange. The healthy group is so large that a small error rate still produces more false alarms than true cases.
The dot count is Bayes’ rule done by hand. Section 2 writes the same count as a formula.
Challenge +8 XPRaise the share of sick people until more than half of the positive results are teal.
Predict first Out of 1000 people, 1% are sick. The test is positive for 90% of the sick and for 9% of the healthy. About how many people test positive?
Pick one guess. The answer appears here.
The book opens with this situation: you know and need . You hold a positive result and want the posterior. The numbers are invented, and the story is added here: means sick and means a positive result.
Section 2
The rule and where it comes from
Here is the rule the book states as Equation 3.42:
Click any coloured symbol to see what it means.
- is the posterior, how probable the cause is after you see the observation.
- is the prior, the probability before you look. In the medical story it is the base rate.
- is the likelihood, how often the observation appears when the cause is present.
- is the evidence, the overall probability of the observation. It makes the posterior add up to 1.
The book states this sum in its running text:
Click any coloured symbol to see what it means.
The evidence looks like an extra demand, but it follows from the other two terms. In the medical story takes two values, so the sum has two terms: the positives from the sick and the positives from the healthy.
Going deeper: the rule from the product rule (added here, not printed in the book)
The rule follows from the product rule. The joint probability of cause and observation can be built in two orders, and both give the same value:
Click any coloured symbol to see what it means.
Set the two outer expressions equal and divide both sides by . The left side becomes the posterior, which is Equation 3.42. Nothing else is hidden in the rule. It is the product rule read in both directions.
Predict first Line A makes 50% of the parts, and 1% of its parts are defective. Line B makes 30%, with 2% defective. Line C makes 20%, with 5% defective. A defective part turns up. Which line most likely made it?
Pick one guess. The answer appears here.
Which production line made the defective part?
prior, the production shareslikelihood, the defect ratesposterior, the answer
The shares are normalized so that the prior adds up to 1. Three things to notice. The likelihood bars do not have to add up to 1, because each one is a defect chance inside a single line. The posterior bars always add up to 1, because the evidence is exactly their sum. And the line with the highest defect rate is not automatically the culprit, because a large share can outweigh a high defect rate.
Names and history
The rule is named after the Reverend Thomas Bayes, who discovered a special case. The general version stated here was found independently by Pierre-Simon Laplace, as the book notes.
Challenge +8 XPMake line A the most probable source while its defect rate stays at 2% or lower.
Challenge +8 XPBreak the rule: find a setting where the evidence is exactly zero. Why can the posterior then not be computed?
Updating with one observation at a time
The book states Bayes’ rule for one observation. Updating over several observations is added here.
The bag holds three coins with heads chances : 0.2, 0.5 or 0.8. You start with no idea which one you hold, so each has the prior .
Predict first You flip the coin in your hand and it comes up heads three times in a row. How probable is it now that it is the coin with heads chance 0.8?
Pick one guess. The answer appears here.
Which coin is in my hand?
coin with heads chance 0.2coin with heads chance 0.5coin with heads chance 0.8
After one observation, the posterior is your new belief, and it becomes the prior for the next.
Record a few heads in a row. The 0.8 coin climbs and the 0.2 coin falls. Each head multiplies the weight of a coin by its heads chance, and the weights are then rescaled to add up to 1. A tail multiplies by one minus the heads chance. Mixed sequences leave the 0.5 coin competitive, because it is never very wrong.
Challenge +8 XPRecord heads until the belief in the 0.8 coin is above 95%.
Two properties are worth checking with the buttons. The order of the flips does not change the final belief, and updating one flip at a time gives the same answer as applying the rule once to the whole sequence. The readout shows both numbers.
Going deeper: why the order does not matter (added here)
Assume the flips are independent once the coin is known. Then the likelihood factors can be multiplied in any order, and the rule for a whole sequence of observations is:
Click any coloured symbol to see what it means.
For a head, is the heads chance. For a tail it is one minus that chance. The product depends only on the counts, so the order cannot matter.
The base rate and the direction of the question
This demo plots the posterior against the base rate , on a log axis.
Predict first The base rate is 1%. The test is positive for 95% of the sick and 5% of the healthy. You test positive. About how likely are you to be sick?
Pick one guess. The answer appears here.
Posterior against base rate
posterior, the answer you wantlikelihood, the printed accuracyyour base rate
Rare conditions take up most of the log axis, where the trouble lies. The violet dashed line is the likelihood, which does not change with the base rate. The teal curve is the posterior, the answer you want. For rare conditions the curve sits far below the line, and that gap is the whole mistake of reading the likelihood as the posterior. Set the false-positive rate to 0% and the curve reaches 100% at every base rate, which shows that false alarms alone create the gap.
Challenge +8 XPKeep the base rate at 1% or lower and make the posterior at least 50%.
This is why the rule matters in machine learning. A model that gives the likelihood of the data under each hypothesis does not by itself say which hypothesis is true. The prior has to be combined with it, and a lopsided prior can overrule a confident-looking likelihood.
Going deeper: the posterior for a yes-or-no cause (added here, not printed in the book)
Take two causes: for sick and for healthy. The evidence then has two terms, and Equation 3.42 becomes:
Click any coloured symbol to see what it means.
The first line keeps the evidence as one symbol, and the second line writes it out as the sum over the two causes. Only three numbers appear: the two likelihoods and the base rate. The dashed line in the demo is the first likelihood, and the teal curve is this formula at every base rate.
What to take away
- and are different quantities. Bayes’ rule converts one into the other.
- The rule is prior times likelihood, divided by the evidence.
- The evidence is the sum over every cause of prior times likelihood, so you do not need it as a separate input.
- The rule is the product rule read in both directions: .
- A small base rate can make a positive result from an accurate test more likely to be a false alarm than a true case.
- The posterior after one observation becomes the prior for the next.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.11 (pages 70–71), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.