How to use this pageClick any colored symbol, in an equation or in a sentence, and it explains itself. Press the buttons, drag the sliders and click the charts. Some answers are hidden until you guess.
, , data and examples , weights and folds functions learned from a fold , , errors and their spread , , hyperparameters
Section 0
Hyperparameters
Think of baking a cake. The batter changes as you mix it, but the oven temperature is set before you start. The parameters, such as the weights , are fitted from the data. The hyperparameters are settings the algorithm does not adjust, such as the polynomial degree and the weight decay strength of Equation 5.18.
Predict first. You fit one model for every degree from 0 to 10 on the training set, and keep the one with the lowest training error. Which degree do you keep?
The reason is short. A degree-3 polynomial is a degree-4 polynomial whose top coefficient is zero, so the larger family fits at least as well. Weight decay works the same way: a smaller removes a restriction, so the training error can only fall as goes to zero.
Going deeper: why a larger family cannot fit worse
Let be the degree-3 polynomials and the degree-4 polynomials. Every member of is also a member of , so the minimum over the larger set cannot be the larger number. This derivation is added here and is not printed in the book.
Click any coloured symbol to see what it means.
The book adds that one learning algorithm can learn the hyperparameters of another, in a nested procedure.
Choosing capacity on the training set
Some settings are fixed by hand because they are hard to optimize. Capacity settings are hyperparameters for another reason. Learned on the training set, they always pick the most flexible model, which overfits (Figure 5.3).
Settings chosen by training error
training errorvalidation errorfitted curve
On the Degree tab the training error never rises as the degree grows, so it always picks the largest degree on offer. On the Weight decay tab it never falls as grows, so it picks the weakest decay. The validation error has its minimum inside the range. Move the slider there, or press Pick by validation error.
Challenge +8 XPOn the Degree tab, move the slider to the degree with the lowest validation error. Then read what the training error would choose instead.
The validation set
The validation error is the score you need, because it is computed on examples the fit never saw. The fix is a validation set: examples the training algorithm does not observe, used only to compare settings. The test set stays out of every choice, so the validation set is cut from the training data. The two parts are disjoint. A typical split uses about 80 percent for training and 20 percent for validation.
Predict first. The model with the lowest validation error is chosen, and its validation error is compared with its true generalization error, averaged over many trials. Which one is lower on average?
Now test your guess. Each trial draws a pool of examples, splits it, picks the best degree on the validation part, and measures three errors of that winner. The demo repeats this 300 times.
Optimism of the validation error
training error of the winnervalidation error of the winnergeneralization error of the winner
The gap grows with the number of candidates, because the lowest of many noisy scores is usually a lucky one. It shrinks when the validation part is larger, because each score is less noisy. The dotted line marks the noise variance, 0.09.
Challenge +8 XPSet the degrees to choose from to 1. With one degree there is no choice to make. Then bring the typical validation error to within 5 percent of the typical generalization error.
Challenge +8 XPChoose from at least 8 candidates and find a setting where the typical generalization error is at least 1.5 times the typical validation error. The readout gives the ratio.
Only after every hyperparameter decision is made should the test set be used, once, to estimate the generalization error. Its error is the honest number to report.
Section 2
The test set and stale benchmarks
A test set can go stale. Reuse it for years, or let a whole field tune against it, and its scores turn optimistic. Picture 100 equally fast runners. Time each one once and publish the fastest time. It looks better than any runner really is.
Predict first. A thousand systems each have a true error of 20 percent. They are scored on one shared test set, and the lowest score is published. What happens on a fresh test set?
Now run the leaderboard. The toy below is added here and is not printed in the book. Each of candidates has the same true error rate , and each is scored on one shared test set of examples. The left chart shows the expected winning score. The right chart shows one simulated leaderboard and the fresh score of its winner.
The winner’s optimism
expected winning score, with the 10 to 90 percent bandtrue error ratethe same winner on a fresh test set
A bigger test set shrinks the effect, because each score is less noisy. More candidates push the winner further below the truth. The fresh score returns the winner to its true rate, on average. Real candidates differ, so the toy is an extreme case. It isolates the mechanism: reuse one test set long enough and it becomes a validation set.
Going deeper: the expected winning score
Let be the number of mistakes of candidate on the shared test set. Each one is binomial with parameters and , and the candidates are treated as independent. The lowest count reaches at least only if every candidate does, so its probability is the chance for one candidate raised to the power . The expected winning score is the expected lowest count divided by , which is a sum of those tail probabilities. This derivation is added here and is not printed in the book.
Click any coloured symbol to see what it means.
Challenge +8 XPFind a leaderboard where the expected winning score is at least 0.05 below the true error rate. Try more candidates, and watch the band.
k-fold cross-validation
One fixed split is fragile when the test set is small, and then claiming that algorithm beats algorithm becomes a guess. The usual remedy is k-fold cross-validation, Algorithm 5.1 in the book. Split the data into non-overlapping folds. On trial , fold is the test set and the rest trains the model. Every example is tested once and used for training times.
Predict first. Five folds on 20 examples. How many examples does each fit train on?
Now run the algorithm one fold at a time on 20 points. The learning algorithm is polynomial least squares of a degree you choose, and the loss is the squared error.
Algorithm 5.1, one fold at a time
held-out squared error of an examplecurve fitted without the current foldtraining points
Press Next fold. The ringed points form the current fold. The violet curve is fitted on all the other points, and the orange segments are the held-out errors. After the last fold every example has one bar, and the mean of the bars is the estimate. A larger gives each fit more data, but it needs more fits. At every fold holds one example, which is leave-one-out cross-validation.
Challenge +8 XPBreak the estimate. Set the degree to 8 and the folds to 20, run every fold, and make the estimate exceed 0.18, twice the noise variance of 0.09.
The book adds a caution: there are no unbiased estimators of the variance of these averaged error estimates (Bengio and Grandvalet, 2004), so approximations are used in practice.
Going deeper: Algorithm 5.1 line by line
Algorithm 5.1: KFoldXV(, , , )
Require: , the given data set, with elements .
Require: , the learning algorithm, seen as a function from a data set to a learned function.
Require: , the loss function, from a learned function and an example to a number.
Require: , the number of folds.
Split into mutually exclusive subsets whose union is .
for from 1 to do
for in do
end for
end for
Return
In symbols, the function trained without fold and the loss of one of its examples are:
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The mean of the returned vector is the estimate , and the three displays above restate the algorithm. They are not numbered in the book.
Click any coloured symbol to see what it means.
Single splits, folds and confidence intervals
One small test set gives a noisy estimate. The demo draws 200 fresh data sets and estimates a polynomial’s error by one hold-out split or by five-fold cross-validation. Each dot is one mistake: the estimate minus the exact value.
Predict first. Which estimate has the narrower spread of mistakes, one hold-out split or five-fold cross-validation?
One split against five folds
one hold-out splitfive-fold cross-validationzero mistake
Most mistakes sit close to zero for both methods. Five folds average the errors over every example, not only the 20 percent in one fold. The price is five fits. The readout uses medians, because one unstable fit can dominate an average.
Challenge +8 XPChoose the examples and the degree so that the five-fold estimate is at least 2 times more precise than the hold-out estimate. The readout compares the typical sizes of the mistakes.
Confidence intervals around the mean error
The algorithm returns one error per example, so their spread measures the uncertainty of the mean. Let be the sample standard deviation of the errors and the standard error of their mean. The 95 percent interval of Equation 5.47 is then:
Click any coloured symbol to see what it means.
The book adds that these intervals are not well justified after cross-validation. It is still common practice, though, to declare algorithm better only if its interval lies below, and does not intersect, the interval of algorithm .
Predict first. Two intervals overlap. What does the common rule do?
Declaring a winner
mean and 95% interval mean and 95% interval
Equation 5.46 in the book uses the true standard deviation in the standard error. Here the sample standard deviation takes its place, which is usual practice. This substitution is added here and is not printed in the book:
Click any coloured symbol to see what it means.
Going deeper: where the square root of the sample size comes from
If the errors are independent with variance , the variance of their mean is the sum of the variances divided by , which is . The standard error is the square root of that. This derivation is added here and is not printed in the book.
The intervals are wide when the errors vary a lot or is small. When and differ little, the intervals overlap and the rule stays silent, even if one of them is truly better. More examples narrow the intervals and produce more verdicts.
Challenge +8 XPChoose different degrees for and so that the rule declares a winner: one interval must lie entirely below the other.
What to take away
- Hyperparameters, such as the degree or the weight decay , are chosen outside the training loop, because training error always prefers the most flexible model.
- A validation set, split disjointly from the training data, compares settings. The test set is used once, after every choice.
- A test set reused for years turns optimistic, and benchmarks go stale.
- k-fold cross-validation tests every example once, returns one error per example, and averages them, at the cost of fits.
- Confidence intervals support comparisons, but after cross-validation they are only approximately justified.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 5.3 (pages 120–123), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.