Section 1
Parameters as random variables
θ, x, w parameters and data p prior, likelihood, posterior μ, ŷ estimates and predictions a, α, λ, τ precisions and widths − log p penalties
How to use this pageClick any colored symbol, in an equation or in a sentence, and it explains itself. Make each prediction before you read the answer, and move each slider before you read the paragraph under it.
Every estimator so far returned one value of the parameters. This page keeps all the possible values instead, each with a degree of belief.
The two schools disagree about what is random. For a frequentist, the true parameter is a fixed unknown, and the estimate is random because it depends on the dataset. For a Bayesian, the observed data are fixed, and is the uncertain quantity.
Predict first. For a frequentist, which quantity is random?
Before any data arrive, the belief about is a distribution, the prior . Practitioners usually choose a broad prior with high entropy, which admits real uncertainty.
Each observation updates the belief with Bayes rule (Equation 5.67):
Click any coloured symbol to see what it means.
Read it as posterior = likelihood × prior / evidence. The likelihood says how well each value of explains the data. The evidence does not depend on , so it only rescales the posterior to total 1.
Predict first. Data add information. After a few observations, is the posterior typically narrower or wider than the prior?
For large , the book notes at the end of Section 5.5, no consistent estimator has a lower mean squared error than maximum likelihood. With few examples the Bayesian view can do better, and a coin shows the mechanism.
The coin has an unknown heads probability . This example is added here and is not printed in the book. Its prior is a Beta distribution with shape numbers and . Each head multiplies the curve by , and each tail by .
Predict first. You start with a uniform prior, so both shape numbers are 1 (). Then you see 8 heads in 8 flips. What is the posterior mean of the heads probability ?
Update beliefs about a coin
priorlikelihood, rescaledposterior
With no flips the posterior is the prior. Every head pushes the curve towards large , every tail the other way, and the spread shrinks as data arrive. The demo multiplies the two curves point by point, then divides by their area, as Equation 5.67 says. Set both shape numbers to 12 and flip heads: the prior already carries 22 imagined flips, so the posterior moves slowly.
Challenge +8 XPGet the posterior mean between 0.45 and 0.55 after at least 10 flips. Click Flip: heads and Flip: tails by hand.
Challenge +8 XPSet both shape numbers to 12, then flip heads until the posterior mean passes 0.6.
Going deeper: the Beta update
Added here, not printed in the book. Multiply the prior by the likelihood of heads in flips and collect the powers of and :
Click any coloured symbol to see what it means.
The result has the same form as the prior with new shape numbers, so the posterior is a Beta distribution with shape numbers and . Dividing by its area gives the posterior, and that is the division the demo performs numerically.
Prediction with the full posterior
The first difference between the Bayesian and maximum likelihood concerns prediction. The Bayesian predicts the next example by averaging over every value of , each weighted by its posterior density:
Click any coloured symbol to see what it means.
Read it as a weighted vote. Each value of says what should look like, and its vote counts in proportion to its posterior density. A wide posterior means many values vote differently, so the prediction is hedged.
Predict first. You see 3 heads in 3 flips. Maximum likelihood predicts that the next flip is heads with probability 1.00. What does a Bayesian with a uniform prior predict?
Predict the next flip
On the first tab the Bayesian line starts far from maximum likelihood and approaches it. On the second, the three priors disagree early and agree later. The book adds that the prior is a source of subjective judgment, and that the integral becomes expensive with many examples.
Challenge +8 XPSet 3 flips seen on the Three predictors tab, then choose the prior strength so the Bayesian prediction is within 0.01 of 0.70.
This cost illustration is added here and is not printed in the book. For a coin the integral is one-dimensional and cheap. For a network with millions of parameters it is not.
Going deeper: why the integral is a posterior mean
Added here, not printed in the book. For a coin, the probability of heads given is itself, so the integral in Equation 5.68 becomes the posterior mean of :
Click any coloured symbol to see what it means.
With the Beta prior from Section 1, the prediction is the fraction on the right. The demo uses the same formula on its Three predictors tab, with .
Example: Bayesian linear regression
The book now works one example through in full: Bayesian linear regression. The prediction is a linear function of the input vector with weight vector :
Click any coloured symbol to see what it means.
Stack the training inputs as the rows of the design matrix and the targets as one vector. Then all predictions come from one product:
Click any coloured symbol to see what it means.
The posterior over is a Gaussian. Predict its width first, then build it.
Predict first. Each weight has a prior standard deviation of 2 (the demo starts at ). What happens to the posterior standard deviation of the slope after all 20 points arrive?
Posterior over a line
true weights (ring)likelihood or priorposterior mean and sampled lines
With no data the posterior equals the prior, and the sampled lines form a random fan. Each data point turns the likelihood into a band of lines that pass near it, and the posterior is the overlap of that band with the prior. Move the prior mean of the slope away from 0: with few points the posterior mean slides towards it.
Challenge +8 XPKeep the prior mean at 0 and the prior standard deviation at 2. Find the smallest number of points for which the posterior standard deviation of the slope is below 0.15, and set the points slider to that number.
Challenge +8 XPWith all 20 points, set the prior standard deviation so the posterior mean slope is about 10% smaller than the least squares slope, between 0.88 and 0.92 times it.
Going deeper: the posterior derivation, Equations 5.71 to 5.78
The likelihood treats each target as a Gaussian around its prediction. The book follows the standard choice and fixes the variance at one. The book writes the training set with the superscript (train), and this page keeps it in the next two equations and drops it afterwards.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The footnote adds that, unless there is a reason to expect correlated weights, the covariance is diagonal, .
Click any coloured symbol to see what it means.
Bayes rule multiplies the likelihood and the prior. A product of Gaussians is an exponential of a sum of quadratics:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The book then defines the posterior covariance and the posterior mean :
Click any coloured symbol to see what it means.
With these, the exponent is a Gaussian in and the posterior is again a Gaussian:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The precision −1 is the data precision plus the prior precision. Precisions add, so every data point makes the posterior narrower. The mean is the data's pull, weighted by , plus the prior's pull.
Going deeper: completing the square
Added here, not printed in the book. Collect the terms of Equation 5.76 that contain . The quadratic part is , which equals . The linear part is , which equals by the definitions of and .
Now expand and compare. It equals . The first two terms match the expression above, so what is left is a constant that does not contain . That constant is the term in Equation 5.77, and the square turns the exponent into the form shown there.
Going deeper: the constant in the exponent
Added here, not printed in the book. The two exponents of Equations 5.75 and 5.77 differ only by a constant that does not depend on . The table draws random weights and computes both exponents, half the quadratic form in each case. Their difference is the same in every row.
The posterior mean and weight decay
The book looks at the posterior mean for a simple prior: zero mean, and covariance a multiple of the identity, and . Then is the prior precision. A large value is a confident belief that every weight is near zero.
Click any coloured symbol to see what it means.
Predict first. The prior precision goes from to (set the slider to 2). What happens to the one-standard-deviation ellipse around the posterior mean?
One estimate, two names
path of the posterior meanposterior mean and ellipsesleast squares (ring: true weights)
The book states the result: the posterior mean is the solution of frequentist linear regression with a weight decay penalty of added to the squared error. That penalty is not numbered in the book. At small the posterior mean sits near the least squares point and the ellipse is wide. Increasing pulls the center towards the origin and shrinks the ellipse.
Challenge +8 XPWith 4 points, move the slider for until the posterior mean slope is half of the least squares slope, within 0.01 of the ratio 0.5.
The book names two differences from the frequentist answer. First, the Bayesian estimate is not defined at , where the prior would be infinitely wide. Second, it also gives a covariance , which shows how likely every other weight vector is.
Going deeper: the ridge formula
Added here, not printed in the book. Write the objective as the negative log posterior, which is half the squared error plus half the prior penalty:
Click any coloured symbol to see what it means.
Setting its gradient to zero gives the normal equations . With , the posterior mean defined above solves the same equations, so the two formulas agree. The readout computes the posterior mean from the matrix inverse and from these normal equations separately. The formula is still defined at when is invertible, and then it equals least squares. The Bayesian posterior is not defined there, because the prior would be infinitely wide.
Maximum a posteriori (MAP) estimation
The full posterior is the most principled answer, but most of its operations are intractable for interesting models. The maximum a posteriori (MAP) estimate is the parameter value where the posterior is highest, a tractable point estimate that keeps some influence from the prior.
Click any coloured symbol to see what it means.
Log-likelihood plus log-prior
log-likelihoodlog-prior (dashed)sum: log-posteriortrue mean 1.5
The evidence does not depend on , so it drops out. What is left is the log-likelihood plus the log-prior, and the logarithm turns the product into a sum.
The log-prior is a downward parabola centered on 0. Adding it to the log-likelihood pulls the peak from the sample mean towards 0, strongly when the data are few and weakly when they are many. Set to 0 and MAP becomes maximum likelihood.
For a unit-variance Gaussian mean, the peak has a closed form:
Going deeper: the closed form for a Gaussian mean
Added here, not printed in the book. The log-posterior is minus half the sum of the squared differences, minus half of times the square of the mean, plus a constant. Setting its derivative to zero gives the sum of the observations equal to times the mean. Dividing by that factor gives the formula:
Click any coloured symbol to see what it means.
Weight decay as a MAP estimate
Predict first. A zero-mean Gaussian prior sits on every weight, with precision . As grows, what happens to the weights?
Take the linear model with a Gaussian prior on its weights, . The book writes the covariance as , the same matrix. Up to a constant that does not depend on , the log-prior is a multiple of the weight decay penalty.
Going deeper: the log of the Gaussian prior
Added here, not printed in the book. The Gaussian density has the exponent and a normalizing constant that does not depend on . Taking the logarithm gives:
Click any coloured symbol to see what it means.
with the number of weights. The last term does not depend on , so it does not change where the maximum lies.
So MAP with a Gaussian prior on the weights is weight decay. The book adds that MAP has less variance and more bias than maximum likelihood.
Predict first. With 5 examples, which prior precision gives the smallest average squared error for the true mean 0.8?
Less variance, more bias
bias squaredvarianceerror = bias squared + variancehighlighted precision
At the estimate is maximum likelihood, the bias is zero and the variance is . As grows, the variance falls and the squared bias rises, so the error has a best moderate precision. The readout compares 2000 simulated data sets with two closed forms, derived in the deeper block and not printed in the book.
Challenge +8 XPMove the highlight to the lowest point of the error curve, within 0.15. The curve is simulated, so read the readout too.
Going deeper: the two closed forms
Added here, not printed in the book. The bias is and the variance is , with .
Priors and penalties
The book states the general rule: many regularized estimators, such as maximum likelihood with weight decay, can be read as the MAP approximation, provided the penalty adds to the objective. The converse does not hold. Some regularizers are not the log of any probability distribution, and some depend on the data, which a prior is not allowed to do. The book cites Nowlan and Hinton (1992) for a mixture-of-Gaussians penalty.
Predict first. Which prior sets small estimates exactly to zero?
The last demo takes one weight whose maximum likelihood estimate is , with standard error . The MAP estimate balances the fit to against the penalty . Switch the prior on the tabs. The Laplace prior is added here and is not in the book. The mixture uses one narrow and one wide Gaussian with scale and .
From prior to penalty to shrinkage
penaltyMAP as a function of zmaximum likelihood, MAP = zyour z
The Gaussian prior shrinks every estimate by the same factor, . The Laplace prior sets small estimates exactly to 0 and moves large ones by a fixed amount. The mixture leaves large estimates nearly alone, so its MAP curve bends sharply and then jumps. With the default scale of 1 that jump is hard to see; a scale of 0.5 shows it.
Challenge +8 XPOn the Mixture tab with standard error 1, set the scale so the MAP for is below 0.5 while the MAP for stays above 2.5.
Challenge +8 XPOn the Laplace tab, set the standard error to 0.5 and the scale to 1. Read the threshold in the readout, then set the observed estimate to within 0.05 of it.
- The MAP estimate maximizes the log-likelihood plus the log-prior (Equation 5.79).
- A Bayesian keeps the whole posterior, and predictions average over it (Equation 5.68).
- With a Gaussian prior and likelihood the posterior is Gaussian, and precisions add (Equations 5.74 to 5.78).
- With a zero prior mean and covariance (1/) I, the posterior mean is the weight decay solution.
- MAP with a Gaussian prior is weight decay, with less variance and more bias than maximum likelihood.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 5.6 (pages 135–140), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.