Section 1
Why probability is needed
How to use this pageClick the colored symbols to see what they mean. Move each slider before you read under it, and commit to a guess in every Predict first box.
Colors: ink counts, teal probabilities, violet estimates, orange errors, gold sizes.
counts and data probabilities estimates errors and losses sizes and small constants
Picture a forecast that says “70% chance of rain”. Nobody asks for a proof, and people plan an umbrella around the number. Machine learning lives in the same kind of world.
Predict first. A reliable processor runs 109 instructions. Each one has a chance of 10-15 of failing. What is the chance that at least one of them fails?
It is about one in a million (10-6), because 109 times 10-15 is 10-6. The demo below sets this case when you press the answer. This processor example is added here and is not printed in the book.
A processor is treated as deterministic: the same input gives the same output, because hardware faults are rare enough to ignore. A model that reads an image or a sensor trace has no such luxury. Its inputs are uncertain quantities, and some are also stochastic, meaning genuinely random rather than merely unknown.
Suppose each step fails with chance , independently of the other steps. The chance that at least one of steps fails is
Click any coloured symbol to see what it means.
This formula is added here and is not printed in the book. Read it from the inside out. is the chance that one step succeeds, and is the chance that all succeed. Failing somewhere is the opposite, . When is small the value is close to . When it is large the value is close to 1.
Rare errors against constant uncertainty
reliable instruction, chance 10-15uncertain prediction, chance from the slider
Set the uncertain prediction's chance with the first slider and the number of steps with the second. The gap between the two curves is the whole story. A billion instructions are safe to ignore, but a hundred uncertain predictions will almost surely go wrong somewhere, so the model has to represent that uncertainty.
Challenge +8 XPSet the two sliders so that the uncertain prediction fails somewhere with a chance between 40% and 60%. The readout gives the chance.
Going deeper: where the formula comes from
This derivation is added here and is not printed in the book. Each step succeeds with chance , and the steps are independent, so all succeed with chance . Failing somewhere is the complement, which gives in the formula above.
The bound follows from the inequality , which holds for every between 0 and 1. It is the union bound: the chance that at least one of several events happens is no more than the sum of their chances. So the straight line is never below the exact value, which makes it a safe estimate when the chances are small.
Three sources of uncertainty
The book names three places where uncertainty comes from. Each one suggests a different remedy, so tell them apart before you try to fix anything.
Incomplete observability means the system is deterministic but some of the variables that drive it are hidden from you. The book uses the Monty Hall game: three doors, a car behind one of them and a goat behind each of the other two.
Predict first. You pick door 1. Before any door is opened, what is the chance that the car is behind door 1?
The car is equally likely to be behind any door, so the chance is one third. Tick Reveal below to see where the car really is. The outcome was fixed all along, and the randomness lives in what you cannot see.
The doors: hidden state, deterministic outcome
win frequency so farone third
With the doors hidden, the outcome looks like a gamble, and the honest summary is a probability. With Reveal ticked, the outcome is a fixed consequence of where the car is and where you pointed. Press Play 100 games a few times and the win frequency settles near one third.
Challenge +8 XPPlay at least 300 games with the demo and get the win frequency within 0.05 of one third.
Going deeper: why one third
This argument is added here and is not printed in the book. Let the car be equally likely to be behind each of the three doors. Your pick does not move the car, so the car is behind the door you picked with chance 1 in 3. In symbols, for a fixed pick.
Inherent stochasticity means the system itself is random. Most interpretations of quantum mechanics treat the motion of subatomic particles this way, and so does a card game with a perfectly shuffled deck. No amount of extra observation removes this kind of uncertainty.
Incomplete modeling means the model discards information it has actually observed. The book's example is a robot that sees the exact position of every object around it, then stores only the grid cell that holds each one. Inside the cell, every position looks the same to the robot.
Predict first. The robot cell width is 2. If you halve the width to 1, what happens to the largest possible error?
The largest possible error is half the width, so it goes from 1 to 0.5. The demo below sets the width to 1 when you press the answer.
Discretizing space creates uncertainty
true positionstored centerrange the object could be in
The robot stores only the center of the cell that contains , so its error is . With cell width , every case obeys:
Click any coloured symbol to see what it means.
This relation is added here and is not printed in the book. The book does not say which point of the cell the robot keeps, so the center is a choice made for this demo. Move the object inside one cell and the stored value does not change at all.
Challenge +8 XPChoose a cell width so that the average error over a cell is 0.1 or less. The readout gives the average.
Going deeper: the average error
This derivation is added here and is not printed in the book. Assume the object is equally likely to be anywhere in its cell. Its distance from the center is then spread evenly from 0 to , so the average error is . The largest error is , as the formula above says.
Simple uncertain rules
Even when the true rule is deterministic, a short uncertain rule can be the better engineering choice. The book compares two rules about birds. “Most birds fly” is cheap to write, maintain and pass on. A rule that lists every exception is expensive, and it is still brittle, because the next unlisted exception breaks it.
Predict first. The first exception written down is young birds that cannot fly yet. How many of the 1000 birds does this exception correct?
It corrects 30 birds, which the demo shows when you press the answer. The flock in this demo is invented for illustration. The counts do not come from the book or from ornithology.
How much is each extra exception worth?
rule with the chosen exceptionsone-line rule, “most birds fly”, 92%
Each exception fixes fewer birds than the one before, while every line costs the same effort to write. A rule with exceptions still declares every unlisted bird a flyer with certainty, and the readout counts how many of those declarations are wrong. The one-line rule makes no such promise. It says 92%, and that is its accuracy, , on the flock.
Challenge +8 XPWrite exceptions until the flock accuracy reaches 97%. Then read how many exceptions that took.
TakeawayUncertainty is sometimes a design choice. A short rule with an honest probability attached can be worth more than a long rule that claims certainty it does not have.
Section 4
Frequencies and degrees of belief
Probability began as a way to count how often repeatable events happen. The standard example is drawing a particular poker hand. An outcome has probability when, if the experiment were repeated endlessly, a proportion of the repetitions would produce it. This is frequentist probability.
Click any coloured symbol to see what it means.
Write for the number of repetitions in which happened, and for the total. The running proportion is then . The limit restates the book's sentence in symbols. It is not numbered in the book. A finite experiment only approximates , and the demo below shows how.
Predict first. You raise the number of trials from 100 to 10000. By what factor does the typical gap between the runs shrink?
The typical gap is the square root shown below. Going from 100 to 10000 trials multiplies by 100, so the gap shrinks by the square root of 100, which is 10.
Click any coloured symbol to see what it means.
Frequencies settle, slowly
run 1runs 2 to 6true probability
Early on the six runs disagree. Later they hug the true value. The typical gap is the spread of the running proportion over many runs, and it has a clean size.
Challenge +8 XPUse 10000 trials or more and make all six runs finish within 0.01 of the true probability. Press Draw six new runs to try another set.
Going deeper: the size of the typical gap
This derivation is added here and is not printed in the book. Each trial produces the outcome with chance , independently, so the count has variance . Dividing by gives the variance of the proportion:
Click any coloured symbol to see what it means.
Its square root, about , is the typical gap that the readout quotes. It shrinks as the square root of the number of trials, which is why a hundred times more trials shrink the gap tenfold.
Degrees of belief
Frequencies do not fit every claim. A doctor says a patient has a 40% chance of flu, but the patient cannot be repeated, so the number is no long-run rate. It is a degree of belief: 1 means certain flu, 0 means certain no flu. Probability read this way is Bayesian probability.
Section 5
Coherent beliefs and logic extended
The book makes a strong claim: if a few common-sense properties of reasoning about uncertainty hold, then Bayesian and frequentist probabilities must obey exactly the same rules. It points to Ramsey (1926) for the details.
Predict first. Your beliefs are 0.6 for flu and 0.6 for no flu. Can a bookmaker make you lose money whatever happens?
A belief is the fair price of a ticket that pays 1 if the event happens. Two beliefs of 0.6 cost 1.2 together, and exactly one of the two tickets pays 1, so you lose 0.2 whatever happens. The demo below sets both beliefs to 0.6.
The book's claim has a concrete consequence: the formulas for a poker player's winning chance are the formulas for a patient's disease chance. The argument below is a special case, added here and not printed in the book. Read a belief as the fair price of a ticket that pays 1 if the event happens. Your beliefs for flu and for no flu are two ticket prices, and exactly one of the two tickets pays 1.
Click any coloured symbol to see what it means.
The loss is zero only when the two beliefs add to exactly 1. Any other pair hands a bookmaker a guaranteed profit, which is a strong hint that reasonable beliefs must obey the rules of probability.
Beliefs that do not add up
your result in that outcome
Challenge +8 XPSet the two beliefs so that they add to exactly 1. The readout must then call them coherent.
Going deeper: why the sign comes out as it does
This derivation is added here and is not printed in the book. If the beliefs add to more than 1, you are too confident. The bookmaker sells you both tickets at your prices, you pay and receive exactly 1 whichever outcome occurs, so you lose . If they add to less than 1, you are too timid. The bookmaker buys both tickets from you, you receive and pay exactly 1, so you lose . Both cases give a loss of , and the guaranteed result is the negative of that.
Probability as the extension of logic
Logic says what must follow once other statements are assumed true or false. Probability says how likely a statement is, given how likely the others are. Logic is the special case where every likelihood is exactly 0 or 1.
Tweety is a bird, an example added here and not printed in the book. The rule “birds fly” holds with strength . If , logic concludes that Tweety flies. If , logic concludes that Tweety does not. In between, logic is silent, and probability still gives a usable number.
From a rule that always holds to a rule that usually holds
what logic can saywhat probability assigns
Challenge +8 XPSet the strength so that probability gives Tweety a 50% chance of flying.
What to take away
- Machine learning always meets uncertain quantities, and sometimes stochastic ones, so it needs probability.
- The book names three sources: inherent stochasticity, incomplete observability and incomplete modeling.
- A simple uncertain rule can beat a complex certain one in cost, upkeep and honesty.
- Frequentist probability counts rates of repeatable events, and Bayesian probability measures a degree of belief.
- Both kinds obey the same rules, and probability extends logic to cases that are not certain.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.1 (pages 54–56), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.