Section 1
Expectation as a balance point
expectation, the balance point
probability weight
spread and co-movement
, fixed coefficients
How to use this pageClick any colored symbol. Inside an equation its meaning appears under the equation; in running text a small note appears. Answer each prediction before the demo reveals the result, and move the controls before you read the text under them.
Throw a fair die many times and average the faces. The average settles near 3.5, although no single throw can show 3.5. Before the demo, predict what a loaded die does to that average.
Predict first. A fair die has , so its expectation is 3.5. Set the tilt to 0.3, which favors the high faces. Does the expectation go above or below 3.5?
Right. Positive tilt puts more weight on the high faces, so the balance point moves right. The expectation is about 4.33 at tilt 0.3.
Not quite. The weight moves toward the high faces, so the expectation rises above 3.5, to about 4.33 at tilt 0.3.
A loaded die and its balance point
The tilt reweights the faces: positive favors high faces, negative favors low ones, and 0 is a fair die. The upper plot puts a weight on each value of , and the triangle marks the expectation. The lower plot throws the die with seeded random numbers and keeps a running average.
For the expectation is . The average of 10,000 throws is .
probability weightexpectationrunning average
Challenge +8 XPWith , tilt the die until the expectation is 4 (within 0.05). Positive tilt favors the high faces.
Challenge +8 XPSwitch to and tilt until the expectation, which is a probability here, is 0.5 (within 0.02).
Read Equation 3.9 in three parts. The sum runs over every value of , gives its weight, and is the measurement there. Picture a see-saw: each value sits on a ruler with a weight equal to its probability, and the expectation is where it balances.
Click any coloured symbol to see what it means.
The definition in the book is the average value of when is drawn from . For a variable with finitely many values it is the weighted sum above, so common values count more.
Predict first. With a fair die, what is the expectation of an indicator that equals 1 when is 5 or 6, and 0 otherwise?
Right. Two of the six faces give 1, so the expectation is 2/6 = 1/3. The expectation of an indicator is the probability of the event.
Not quite. The indicator is 1 on two faces out of six, so the expectation is 2/6 = 1/3, which is the probability of rolling 5 or 6.
In the lower plot of the demo the running average wanders at first and then settles on the expectation. (The long-run-average reading is added here as the meaning of the definition. The law of large numbers is not discussed on these book pages.)
Section 2
Expectation of a continuous variable
A continuous variable has no list of values to add up, so the sum becomes an integral. Predict first, before the formula appears.
Predict first. A Gaussian with mean 0.5 and standard deviation 0.8. What is ?
Right. 𝔼[x2] = μ2 + σ2 = 0.25 + 0.64 = 0.89. The square of the mean is only part of it: the spread adds the rest.
Not quite. For a Gaussian 𝔼[x2] = μ2 + σ2 = 0.25 + 0.64 = 0.89. Neither 0.25 (the mean squared) nor 0.64 (the variance alone) is the answer.
The density gives probability per unit length, so a thin slice of width around carries probability . Equation 3.10 replaces the sum with an integral:
Click any coloured symbol to see what it means.
Read the integral as a sum over infinitely many thin slices: each slice contributes its probability times the measurement .
An integral as a sum of thin slices
The density is a Gaussian (introduced in Section 3.9 of the book) with the mean and standard deviation set by the sliders. The upper plot is . The lower plot is , and the area under it is the expectation. Cut the axis into slices and add the rectangles: as grows the sum approaches the integral.
Slices: . Sum of : , which should be 1. Sum of : . Exact expectation: . Error: .
Challenge +8 XPPick and , then find a slice count at most 10 that gets the error below 0.001. Try nine and ten.
With 2 slices the rectangles are crude and the sum is off; with 40 it is accurate to several digits. For = the exact answer is . For = it is 2 + 2, and for it is cos() e−2/2. (These closed forms are standard Gaussian integrals, added here as a check.)
Going deeper: the notation shortcuts
The book allows shorter notation. When the distribution is clear from the context you may write only the variable, as in . When the variable is clear too, you may write . By default averages over all the random variables inside the brackets, and when there is no ambiguity the square brackets may be dropped.
Linearity of expectation
Expectations are linear. The average of a weighted sum of functions is the same weighted sum of their averages, as long as the weights are fixed (Equation 3.11):
Click any coloured symbol to see what it means.
Here and are fixed numbers, while and are functions of . Linearity is one of the most used facts in the book: it lets you average a complicated expression term by term.
Predict first. A fair die has . What is ?
Right. = (1 + 4 + 9 + 16 + 25 + 36)/6 = 91/6 ≈ 15.17. It is larger than ()2 = 12.25 by the variance.
Not quite. = 91/6 ≈ 15.17, not ()2 = 12.25. The square is taken before the average.
Combine first or average first?
The same loaded die as before, with = and = . The upper bar averages the combination + directly. The lower bar combines the two separate averages. Tick the box to let change with .
Average of the combination: . Combination of the averages: .
Also = , but ()2 = .
average the combinationcombine the averages
Challenge +8 XPUse a fair die (tilt 0) with a constant and at least 0.3 in size. Choose so the combined average is within 0.02 of zero.
The book attaches one condition: and must not depend on . The reason sits in Equation 3.9. A constant can be pulled out of a sum, but a coefficient that changes with cannot.
Going deeper: why products do not pass through the expectation
Linearity covers sums only: products and functions of an average do not pass through the expectation. So differs from ()2 by the variance, which is the subject of the next section. (These points are added here and are not printed in the book.)
Variance and standard deviation
Two experiments can share an average and still differ in spread. The variance measures spread as the expected squared distance from the expectation (Equation 3.12):
Click any coloured symbol to see what it means.
Work from the inside out. is the balance point, and − is the distance of one outcome from it. Squaring punishes large deviations, and the outer averages the squares. The square root, , has the units of .
Predict first. You double the spacing between the possible values. By what factor does the variance change?
Right. Doubling every distance doubles each deviation, and squaring makes the variance grow by 4. This holds for any weights, so the concentration does not change the factor.
Not quite. Doubling doubles each distance from the mean. Squaring then multiplies the variance by 4, not 2.
Variance as an average height on a parabola
Seven possible values spaced apart around a location , with probabilities set by the concentration (0 gives equal probabilities). Each value sits on the curve , drawn as a dot sized by its probability. The variance is the probability-weighted average height of the dots.
Mean = . Variance = , standard deviation . Check: − = − = .
variancemean ± std. dev.
Challenge +8 XPSet the concentration to 0, then the spacing, so that the variance is 4 (within 0.05).
Three facts to test with the sliders. The location slides everything sideways and leaves the variance unchanged. Double and the variance is multiplied by four, because distances are squared. A higher concentration pulls probability onto the middle dots, which sit low on the parabola, so the variance drops.
An equivalent form, often easier to compute, follows from linearity. (This derivation is added here and is not printed in the book.)
The variance is therefore minus , which is never negative.
Going deeper: the shortcut form of the variance
Start from Equation 3.12 and write the square out: ( − )2 = 2 − 2 + 2. Apply the expectation to each of the three terms. The constant comes out of the expectation, because it is a fixed number, so the middle term becomes −22. The last term is a constant too, so its expectation is itself. Adding up gives E[2] − 2, which is the shortcut form of Equation 3.12. (This derivation is added here and is not printed in the book.)
Covariance and correlation
Variance looks at one variable. Covariance compares two: it asks whether, when one variable is above its mean, the other tends to be above its own mean too (Equation 3.13):
Click any coloured symbol to see what it means.
For each outcome multiply the two deviations from the means, then average. Both above or both below gives a positive product, and one above with the other below gives a negative one. High absolute values mean that both variables move far from their means at the same time.
Predict first. You measure in millimeters instead of meters, so every value grows a thousandfold. What happens to and to the correlation?
Right. Covariance carries the scales of the variables, while correlation divides the standard deviations out. The book makes the point that covariance carries scale (page 61). The millimeter example itself is added here.
Not quite. Covariance grows with the units, because it carries the scale of the variables. Correlation divides the standard deviations out, so it does not change.
Quadrants and the sign of the covariance
250 seeded points. A point is teal when its product of deviations is positive and orange when it is negative. Drag the scale sliders to see the covariance follow the spread while the relation stays put.
Covariance: . Correlation: . % of the points are in the teal quadrants. Population value = . The gap to the sample covariance is sampling noise: 250 points give an estimate that varies around the population value.
product of deviations > 0product of deviations < 0
Challenge +8 XPMake the covariance above 1 while the correlation stays below 0.5. Spread the axes and keep the link weak.
With = and = , Equation 3.13 becomes Equation 3.12, so variance is the covariance of a variable with itself. Correlation divides the scales out. The book names it but does not print the formula, so the display below is the standard definition.
The covariance changes in proportion to , while the correlation stays put.
Section 6
Zero Covariance and Dependence
Take uniform on [−1, 1] and attach a random sign , equal to +1 or −1 with probability one half each, giving .
Predict first. Predict first: is zero, and are and independent? Take with uniform on [−1, 1].
Right. The covariance is zero, but x fixes the size of y, so the two are dependent. Covariance only sees linear dependence, and that is the blind spot.
Not quite. The covariance is zero because the signs cancel on average, yet x completely determines the size of y, so the two are not independent.
Independence implies zero covariance, but the reverse is false. The book states that independent variables have zero covariance (page 61), and that a nonzero covariance proves dependence. Zero covariance only excludes linear dependence, because independence also excludes nonlinear relationships.
Four relationships, one blind spot
300 seeded points with uniform on [−1, 1]. The first check is the correlation of and , which is the covariance rescaled. The second check is the correlation of and , an extra probe chosen for this demo to expose nonlinear links. It is not a general test of independence.
= , correlation .
Correlation of and : .
The book's example, worked in one line below (added here and not printed in the book): and are independent, so the first term factorizes, and = 0 and = 0 make both terms vanish.
Going deeper: why the covariance of x and sx is zero
Let s be +1 or −1 with probability 1/2 each, independent of x, and let y = sx. Then Cov(x, y) = E[xy] − E[x]E[y] = E[s x2] − E[x]E[sx]. Because s and x are independent, E[s x2] = E[s]E[x2]. The average of s is 0, so this term vanishes. Also E[x] = 0 for x uniform on [−1, 1], so the second term vanishes too. For that x the value of E[x2] is 1/3, which does not matter, because it is multiplied by 0. The covariance is exactly zero. (This derivation is added here and is not printed in the book.)
The sign averages to 0 at every size of , so the products cancel on average. The magnitude check catches the dependence: and are strongly correlated for and for , but not for independent variables. A feature can have zero correlation with a target and still be informative, so a model that looks only at linear correlations will miss it.
Section 7
The covariance matrix
With components you need every pairwise covariance. The covariance matrix collects them in an by table (Equation 3.14):
Click any coloured symbol to see what it means.
Row and column hold the covariance of component with component . Since covariance is symmetric, the matrix is symmetric. Its diagonal holds the variances (Equation 3.15):
Click any coloured symbol to see what it means.
For two components the whole matrix is:
A covariance matrix and the cloud it describes
The ellipses are the one and two standard deviation contours of the Gaussian with this covariance matrix. Their tilt is set by the off-diagonal entry. The readout compares the matrix estimated from 300 points with the true one.
True matrix above. From the 300 points: variances and , covariance (the same in both off-diagonal places, since the matrix is symmetric).
Three things to notice. The diagonal holds 2 and 2, the squared widths along the two axes. The off-diagonal entry is , which is zero when = 0, and then the ellipse lies along the axes. The matrix estimated from 300 points matches the true one only roughly, because a finite sample carries sampling noise.
With more components the picture is the same, drawn as a grid.
A 4 × 4 covariance matrix
The variables are built from independent noise: = , = + , = − + 0.5, and = . The slider sets . The heat map shows the 4 by 4 sample covariance from 400 seeded draws. Outlined diagonal cells are variances.
= (expected about a2 + 1 = ). = . = = .
positive covariancenegative covariance
Challenge +8 XPSet at least 2, then make above 4.
The matrix is symmetric, and its diagonal holds the variances. is about 1, and = 2 + 1 grows with 2. Sampling noise moves the estimates: at = 2 the variance of reads 4.60 against an exact 5, and reads about −0.9 against −1. The row and column of are close to zero off the diagonal, since shares no noise with the others.
What to take away
- The expectation is the probability-weighted average of : a sum for discrete variables (3.9), an integral for continuous ones (3.10). It is the balance point of the distribution of .
- Expectation is linear for fixed coefficients (3.11). It does not commute with products or nonlinear functions.
- Variance (3.12) is the expected squared distance from the expectation. It equals minus , ignores location, and scales with the square of the scale.
- Covariance (3.13) measures linear co-movement and carries the scale of the variables. Correlation divides the scales out.
- Independent variables have zero covariance. Zero covariance does not imply independence.
- The covariance matrix (3.14, 3.15) is symmetric with the variances on its diagonal.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.8 (pages 60–62), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.