Determining Whether to Gather More Data

Section 1

Training error first

How to use this page

Click the colored symbols. Clicking a symbol explains it and lights up the same symbol in the other equations. Commit to a guess before each reveal. Each demo shows the book’s rule once you have chosen.

data, models and sizes errors and gaps predictions and weights the main object penalties and targets

After the first end-to-end system works, you need to decide what to do next. Many novices try a new algorithm every day. The book says that gathering more data is often a better use of time. The question is when. The answer depends on two errors, the error on the training set and the error on a test set, and on the gap between them.

Predict before you read the rule. The training error is 0.40, your target is 0.05, and the test error is 0.45. Which step do you take first?

Pick a guess.

The decision dial

Training error E_train

Test error E_test

Target error E*

A gap of 0.10 or more counts as much worse. That cut-off is added here for the demo and is not printed in the book.

Set the training error first. If it is above your target, the learning algorithm is not using the data it already has. The book is clear that there is then no reason to gather more data. Enlarge the model, tune the learning rate, and only if those fail, suspect the data itself.

Challenge +8 XP

Keep the training error at or below the target, then raise the test error until the dial says to gather more data.

Check

Raise the training error above the target. The dial now says to enlarge the model or tune the optimizer, whatever the test error is. Gathering data comes last in that branch, not first.

Section 2

Test error, the gap and regularization

If the training error is acceptable, measure the test error. If the test error is acceptable too, there is nothing left to do. If the test error is much worse than the training error, the book says gathering more data is one of the most effective solutions. Before you commit, the book asks you to weigh its cost and feasibility, the cost of other ways to cut the test error, and how much data is likely needed. A simpler alternative is to reduce the size of the model or improve regularization.

The generalization error is the error on new examples, and the test set stands in for it. Each error is a mean squared error over the examples in its set. Here is the matrix of basis values for the inputs, and holds the true outputs.

Click any coloured symbol to see what it means.

(The mean squared error and the gap are added here as the measures used in the demos. The book does not print these two formulas.) The gap is , the test error minus the training error. A large gap says the model has learned details of its training set that do not carry over.

A penalty instead of more data

Penalty, as a power of ten λ

training errortest error

The model here has degree 9 and 64 training examples, and only the penalty changes. A penalty adds the weight size to the cost. The fit is then the solution of a regularized least squares problem, which is added here and not printed in the book:

Click any coloured symbol to see what it means.

(This ridge formula is added here to explain the demo. It is the standard form, and the book does not print it in this section.) Raise and the gap shrinks, but the training error rises. A small gap alone does not mean a good model.

Challenge +8 XP

With 64 examples, change only the penalty so that the gap falls below 0.02 while the test error stays below 0.10.

Check

At the default penalty of 0.0001 the gap is about 0.028. At a penalty of 100 the gap is nearly zero, but both errors are above 0.3. A gap of zero from underfitting is not a success.

Section 3

Learning curves on a doubling scale

The book suggests plotting how the error changes with the size of the training set. These plots are called learning curves. By extrapolating the curve, you can estimate how many more examples would reach a target. Because adding a small fraction of the examples usually has no noticeable effect, the book recommends trying sizes on a logarithmic scale, doubling the number between experiments. The book’s figure 5.4 is one such plot.

Predict before you open the demo. The model has degree 9 and the penalty is at its smallest. What is the test error with 512 examples?

Pick a guess.

Two errors against the number of examples

Degree of the model deg

Penalty, as a power of ten λ

training errortest error

The curves flatten. Training error rises toward the noise in the data, and test error falls toward it. The gap between them shrinks as examples are added. The noise set here is a fixed sequence, so every run gives the same numbers. A real curve would be averaged over several random samples.

Check

Drop the degree to 3 and the gap at 512 examples falls to about 0.002. Both curves then sit near 0.085. The noise in this toy has variance 1/12, about 0.083, so that is close to the floor the noise sets. More data would gain little, and the book’s other options come first.

Section 4

Doubling, not adding a little

The ladder below uses sizes from 8 to 512 examples, each double the last. The model has degree 9 and the penalty stays at 0.0001 throughout. Each bar is the test error at one rung. Read the trend, not a single step.

Predict before you move the slider. Going from 64 examples to 128, the test error falls by about how much?

Pick a guess.

Choose the rung

Training set size n

test error at each sizechosen size

The 0.01 band is a choice made for this demo, and the book does not print a band. The book’s general point is that adding a small fraction of the examples usually has no noticeable effect on the error.

Challenge +8 XP

Choose the smallest training set whose test error is within 0.01 of the error at 512 examples.

Check

The rung at 64 examples is about 0.012 above the rung at 512, outside the 0.01 band, though not by much. At 128 examples it is about 0.001 above, inside the band.

Section 5

The gather-or-not game

Each round gives you the training error, the test error and a note about the data. Choose the step the book’s rules point to. The rounds follow the order the book gives: training error first, then the test set, then regularization, then data.

Five rounds

0 of 5 right

The game uses the book’s rules. The rounds are short scenarios written here, and the answer to each is the step the book’s order implies. The biopsy round illustrates the book’s medical example, where gathering data may be costly or infeasible.

Challenge +8 XP

Answer all five rounds of the game correctly.

Summary

Check the training error first. A poor training error means more data will not help yet.
When the gap between test and training error stays large after regularization is tuned, gather more data, unless cost or feasibility rules it out.
Try a smaller model or stronger regularization before gathering, and measure again.
Plot learning curves on a doubling scale, and read the trend rather than a single step.

Final quiz

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 11.3 (pages 426–427), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.