How to use this pageClick any colored symbol, in an equation or in a sentence, and it explains itself. Press the buttons, drag the sliders and click the charts. Some answers are hidden until you guess.
, , , true values and data , , estimates bias, MSE, error bias and error Var, SE variance and spread , , , sample size and constants
Section 0
One dataset, many possible answers
Every number a learning algorithm produces comes from a finite sample. Collect a different sample and the same algorithm returns a different number.
Picture an archer shooting at a target. The bullseye is the true value, and each arrow is the answer one dataset produces. A tight group away from the bullseye is bias. Scatter around the bullseye with no offset is variance.
Predict first. Ten arrows form a tight group about one meter to the left of the bullseye. Which error is large?
Bullseye, bias and spread
arrow (one estimate)center of the groupbullseye (true value)
Move the two sliders. The offset of the group's center from the bullseye is the bias, and the width of the cloud is the spread. The bar underneath measures the average squared distance from an arrow to the bullseye. It always splits into two parts: the squared offset of the center, and the average squared distance of the arrows from their own center. Equation 5.54, in Section 5.4.4 of the book, states this split for estimators, and the readout checks it for every setting.
Challenge +8 XPMake the squared offset of the center at least four times the variance. A small spread with a clear offset does it.
Point estimation
Point estimation gives one best guess for a quantity of interest, such as the weights of a linear regression or a whole function. The book puts a hat on an estimate, so estimates .
Suppose we collect independent and identically distributed points . A point estimator, or statistic, is any function of the data:
Click any coloured symbol to see what it means.
The coin flip makes this concrete. The coin lands heads with probability , and an estimator is any recipe that turns the flips into a guess.
Going deeper: the frequentist view
The book takes the frequentist view. The true is a fixed but unknown number, and the data come from a random process. Every function of the data is therefore random too, so is a random variable. The subscript records how many points the function received. A good estimator returns a value close to the true that generated the data.
Predict first. As the number of flips grows, which recipes come closer to the true coin?
An estimator is a random variable
how often each estimate occurstrue average of the estimates
The four recipes are all legal estimators under Equation 5.19. The sample mean averages every flip. The single flip answers 0 or 1. The constant ignores the data. The add-one rule returns (heads + 1) divided by . The sample mean and the add-one rule tighten around the truth as grows. The single flip never improves, and the constant is right only when the coin is exactly fair.
Challenge +8 XPOn the Sample mean tab, make the standard deviation of the estimate 0.06 or less. Raise the number of flips until it is. Make your guess above first.
Function estimation
The same idea covers an input and a target , with noise . The goal is a model that approximates . A fitted function is a point estimator in function space.
A fitted curve is a random function
true fit to the highlighted dataset24 other datasets
Each dataset is a fresh draw of noisy points from the same curve , so the gray curves show the distribution of the estimator. A low degree gives tight but wrong curves. A high degree gives curves that are right on average but erratic. The readout measures the bias and variance at .
Going deeper: earlier sections as estimators
The linear regression of Section 5.1.4 and the polynomial regression of Section 5.2 can be read as estimating a weight vector , or as estimating a function.
Bias
The bias of an estimator measures how far its average answer sits from the truth:
Click any coloured symbol to see what it means.
Here the expectation averages over datasets of size . An estimator is unbiased if its bias is 0. It is asymptotically unbiased if the bias fades as the data grow.
Example: the Bernoulli distribution
A Bernoulli variable is a coin flip: it equals 1 with probability and 0 otherwise. Its probability mass function is
Click any coloured symbol to see what it means.
The natural estimator is the fraction of ones in the data, the mean of the samples:
Click any coloured symbol to see what it means.
Predict first. Does the add-one rule, which returns , have zero bias?
The demo finds the exact distribution of each estimator, with no simulation. For a given , the estimator takes only the values 0, 1/, 2/ up to 1, each with a known binomial probability. The sum of value times probability is the exact expectation.
The exact distribution of an estimator and its expectation
probability of each valuetrue expectation of the estimator
Challenge +8 XPOn the Add-one rule tab, make the exact bias at least 0.1 in size. Pick a probability and a number of flips that push the rule away from the truth. Make your guess above first.
For the sample mean the bias is exactly 0. The add-one rule has exact bias , which shrinks as grows, so the rule is asymptotically unbiased.
Going deeper: why the Bernoulli sample mean is unbiased
Substitute the estimator into Equation 5.20. The expectation of a sum is the sum of the expectations, and the constant comes out of the expectation:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The expectation of one flip is a weighted sum over its two possible values, each weighted by its probability:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The term with vanishes because of the factor , and the term with equals . The average of copies of is again, so the bias is .
Example: the Gaussian mean
For Gaussian data with mean and variance , the density is
Click any coloured symbol to see what it means.
The estimator is the sample mean, Equation 5.30:
Click any coloured symbol to see what it means.
Going deeper: the Gaussian steps
Its bias follows the same steps as in the Bernoulli case. The one fact needed is that each sample has expectation :
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Example: two estimators of the variance
The first is the sample variance, the average squared distance of the samples from the sample mean :
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Equations 5.38 and 5.39 give its expectation:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Subtracting from Equation 5.39 gives the bias:
Click any coloured symbol to see what it means.
So the sample variance underestimates on average.
Predict first. You average the divide-by- sample variance over many datasets of points, with true variance 1. Where does the average land?
Bias of the variance estimators
estimates over 4000 datasetstrue valueaverage of the estimates
divide by divide by predicted bias
Select the sample mean first. Its average sits on the true value for every . Then choose divide-by- with a small : the average lands clearly below the true value, and the lower chart shows its bias following . Divide by and the bias wanders around zero, within simulation noise.
The unbiased sample variance divides by instead, Equation 5.40, and its expectation is exactly .
Going deeper: the unbiased divisor
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Measure the samples from the sample mean instead of the true mean. The identity below is exact, and the expectation of the correction term is . This derivation is added here and is not printed in the book.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The book ends this subsection with a caution: unbiased estimators are clearly desirable, but they are not always the best. Section 5.4.4 of the book shows why.
Variance and standard error
Variance measures how widely the estimates scatter as the dataset is resampled. The variance of an estimator is the variance of :
Click any coloured symbol to see what it means.
Its square root is the standard error, . A low-variance estimator gives nearly the same answer on every dataset.
Standard error of the mean
For the sample mean of independent samples with true variance , the standard error is
Click any coloured symbol to see what it means.
Going deeper: the standard error derivation
The derivation rests on two facts. The variance of a sum of independent variables is the sum of their variances, and a constant multiplies a variance by . The sum of variances is , and the factor turns this into . The square root gives . This derivation is added here and is not printed in the book.
In practice is unknown, so the standard error is estimated from the data. Both the square root of the sample variance and the square root of the unbiased variance underestimate on average, though the second is the smaller underestimate.
Example: the Bernoulli estimator
A single flip has variance , so the sample mean of flips has variance
Going deeper: the Bernoulli variance
The book repeats the calculation for the Bernoulli sample mean. The steps are:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The variance of this estimator falls as .
Predict first. The standard error of a mean of 100 Bernoulli flips is about 0.05 when the coin is fair. How many flips give a standard error of 0.025?
Variance of the Bernoulli estimator
formula simulation, 3000 datasets
On the second chart the standard error is a straight line of slope −1/2 on log-log axes, because is a power law. To halve the error one needs four times the data.
Confidence intervals and comparing algorithms
The sample mean of the test-set losses estimates the generalization error. By the central limit theorem, its 95% confidence interval is
Click any coloured symbol to see what it means.
The book uses 1.96 because the interval is centered on the estimate and extends 1.96 standard errors either side, under the normal distribution.
Going deeper: why 1.96, and what the interval means
This explanation is added here and is not printed in the book: 95% of a Gaussian's probability lies within 1.96 standard deviations of its center. The guarantee is about the procedure. Over many repeated datasets, about 95% of such intervals would contain the true mean. It is not a claim about the one interval in hand.
Predict first. You build 95 percent intervals from 40 datasets of 3 points each. About how many of the 40 intervals contain the true mean?
Does the 95% interval really cover the truth?
interval contains interval misses true mean
Each row is one dataset of Gaussian samples with true mean 0 and true standard deviation 1. The interval uses the standard error estimated from the data, as in practice. With few samples the covered fraction falls below 95%, because the estimated standard error varies from dataset to dataset and is on average a little too small. The covered fraction is about 81% at and about 90% at , and the deficit vanishes as grows.
Challenge +8 XPRaise the number of samples until at least 93 percent of the 4000 simulated intervals cover the true mean. The readout gives the fraction. Make your guess above first.
Machine learning papers often declare algorithm better than algorithm when the upper end of the interval for ’s error lies below the lower end of the interval for ’s. The next demo gives the two algorithms fixed true error rates and lets you vary the size of the test set.
Two algorithms, one test set
Challenge +8 XPChoose a test set size where the rule declares better in at least 90 percent of the 200 repeated test sets. Keep the gap between the two algorithms as it is.
With a gap of 0.05, a test set of a few hundred examples rarely separates the intervals, even though really is better. A few thousand examples usually do. A failure to separate the intervals does not prove that the algorithms are equal, and the rule says nothing about how much better one is.
Section 4, book 5.4.4
Trading off bias and variance to minimize mean squared error
Often a choice must be made between an estimator with more bias and one with more variance. The book names cross-validation, explained in Section 5.3 of the book, and the mean squared error (MSE) of the estimates:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
MSE is the expected squared distance between the estimator and the true parameter. Equation 5.54 splits it into squared bias and variance, so a little bias can buy a large cut in variance.
Predict first. Estimator has bias 0.3 and variance 0.1. Estimator is unbiased, with variance 0.5. Which has the smaller mean squared error?
Going deeper: why the cross term disappears
Split the square around the mean of the estimator. Then expand it. The first term of the expansion is the variance, by Equation 5.45. This derivation is added here and is not printed in the book.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Capacity and the U-shaped curve
Capacity, here the polynomial degree , sets how complicated a function the model can follow. Raising it lowers the bias of the fitted function and raises its variance, because a flexible model also follows the noise in one sample.
Predict first. The training set shrinks from 60 points to 10. In which direction does the degree with the lowest error move?
Simulated illustration of the U-shape (not the book figure)
bias² (dotted)variance (dashed)generalization error = bias² + variance + noisechosen degree
The book’s Figure 5.6 is a sketch. The demo above simulates the same U-shape, added here, by fitting polynomials of degree 0 to 9 to 300 datasets drawn from sin(3x + 0.8).
At degree 0 the model cannot bend, so bias dominates: the underfitting zone. At the highest degrees variance dominates: the overfitting zone. Shrink the training set and the variance climbs, so the best degree moves left.
A biased estimator can win
Equation 5.54 explains why the book warns that unbiased estimators are not always the best. Compare the divisors equal to (unbiased) and (Equation 5.36).
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Predict first. Which divisor gives the smallest mean squared error for the variance of Gaussian data: , , or ?
Which divisor gives the smallest mean squared error?
bias²varianceMSE = bias² + variancechosen divisor
Challenge +8 XPChoose a divisor whose MSE is at least 10 percent below the unbiased divisor . Set the divisor slider, not the sample size. Make your guess above first.
Going deeper: the closed form for the divisor family
For Gaussian data, the sum of squared deviations divided by follows a chi-squared distribution with degrees of freedom. Its mean is and its variance is . Dividing by gives the bias and variance in closed form. This computation is added here and is not printed in the book.
The unbiased divisor is not the minimum. Its bias term vanishes, but its variance is the largest of the three. The divisor shrinks the estimate, which cuts the variance by more than the squared bias adds. The minimum sits at , where the MSE equals . The gap is largest for small and fades as grows. An estimator should be judged by both kinds of error together, not by bias alone.
Section 5, book 5.4.5
Consistency
So far the dataset size was fixed. We also want the estimates to converge to the true value as it grows:
Click any coloured symbol to see what it means.
The symbol plim means convergence in probability. For every tolerance , the chance of missing the truth by more than must go to 0 as grows:
An estimator with this property is consistent. Consistency is about large datasets, and unbiasedness is not required.
Take the single sample as an estimator of the Gaussian mean . It is unbiased, but it never improves as grows.
Predict first. Use only the single sample as the estimator. As the dataset grows, does its chance of missing the truth by more than a fixed tolerance fall?
Convergence in probability
sample meansingle samplemean times exact value for the sample mean
The data are Gaussian with true mean 1, from 300 simulated experiments per point. The sample mean concentrates on the truth, and the single sample never does. The shrunken mean is biased at every size, yet consistent, because its shrinkage factor tends to 1.
Going deeper: maximum likelihood
Consistency is one reason maximum likelihood, the topic of the next module, is trusted. The book states that the maximum likelihood estimator is consistent under appropriate conditions, and it lists those conditions.
- Estimator (Eq. 5.19): any function of the data, so its estimate is a random variable.
- Bias, variance and MSE (Eqs. 5.20, 5.45, 5.54): MSE = bias² + variance, so a biased estimator can win.
- Consistency (Eq. 5.55): the estimate converges in probability to the truth, with no need for unbiasedness.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 5.4 (pages 122–131), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.