Example: Multi-Digit Number Recognition

Choose the metric

A map with wrong house numbers misleads its reader, so the team that transcribed Street View house numbers set a high bar. The book's target was human-level accuracy of 98 percent. The system can only reach that bar by refusing the inputs it is unsure about. Refusal costs coverage, the share of inputs the system answers. The team therefore chose coverage as the main metric to push up, with accuracy held at 98 percent.

The system refuses an input when its confidence falls below a threshold . These two definitions are added here and are not printed in the book, which describes them in words:

A higher threshold refuses more inputs. Accuracy on the answered inputs tends to rise, and tends to fall. Your task is to find the threshold that gives 98 percent accuracy and keeps as much coverage as possible.

The toy population has 400 inputs, generated from a fixed seed. Most are easy, with high confidence and high accuracy. The rest are hard. The numbers describe this toy and are not the Street View data.

The book reports that as the convolutional network (a network that learns image filters) improved, the threshold could come down while accuracy held at 98 percent, and coverage eventually passed the goal of 95 percent.

Score a whole address

An address has several digits, and each digit is one classification. The first design gave each position its own softmax output, which turns class scores into probabilities that add up to one. Each output was trained as a separate classifier. The book calls this a simple baseline. The system still needs one number for the whole sequence, because refusal applies to the whole address.

The book says its first definition of the sequence probability was ad hoc. It multiplied the softmax outputs of the digits together. The product is written below, with the same meaning as the book's words. The log form is added here and is not printed in the book.

The book then replaced that rule with a specialized output layer and a cost function that computes a proper log-likelihood. That change made the refusal rule work much more effectively. The refusal rule itself is the same idea as before:

Each digit below certainty multiplies the product by a number below one. Three digits at 90 percent give 72.9 percent, which is below 80 percent, even though every digit looks confident. This reading is added here and is not printed in the book.

Compare train and test error

The book's first check is whether the problem is underfitting (the model is too simple for the data) or overfitting (the model fits its training examples but not new ones). When the team measured the system, training error and test error were nearly identical. The book reads this as underfitting or a problem in the training data, not overfitting.

The demo below uses a toy error curve. The training error is 30 × 0.7 percent, and the test error is half a point higher. The model size is the learner's control. A checkbox adds 8 points to both errors, which is what a fault shared by both sets would do.

The book's conclusion is that when training and test error are roughly equal, the remaining deficit is underfitting plus a few problems in the dataset. The next section looks at the worst errors, the check the book used to find the cropping problem.

When training error is low and test error is high, the pattern is overfitting. This toy does not produce that pattern.

Read the worst mistakes

The book's debugging strategy is to visualize the model's worst errors. In this project those were the incorrect transcriptions the model gave the highest confidence. Most of them were addresses cropped so tightly that digits were lost. For example, a photo of the address 1849 might be cropped to show only 849, as in the book.

To measure how much of the first digit a crop removed, define the cut share. This definition is added here and is not printed in the book.

The toy has two kinds of mistake: tight crops that cut the first digit, and a smaller group of ambiguous digits with a clean crop. Both kinds can be confident, so the share of cut digits among the confident mistakes changes from draw to draw. The same plot made from real data is the check that matters, because only the data can say which cause dominates. The reading that a cause in the input explains a group of confident errors is added here and is not printed in the book.

The book notes that this could have been fixed by spending weeks improving the detector that chooses the crop. Instead, the team widened the crop. A pattern like this points to the input pipeline rather than the network. This reading is added here and is not printed in the book.

Widen the crop, then tune the model

The fix was to make the crop systematically wider than the detector predicted. The book says this single change added ten percentage points to coverage. The crop margin is the extra width added on each side. In this toy a digit is kept when its offset from the crop center is within the margin plus half a digit width:

The rule above is added here and is not printed in the book. The offsets in the toy are spread evenly over 1.6 digit widths on each side. Move the margin and watch which addresses keep their first digit.

The book's team also tuned hyperparameters, the settings chosen before training, such as the model size. The last few percentage points came from making the model larger while keeping its computational cost within a limit. Train and test error stayed roughly equal through this phase, so the remaining deficit was underfitting and a few dataset problems.

What to take away

• Choose the metric from the business goal. Here the goal was 98 percent accuracy, with coverage as the metric to push up.
• Score a whole sequence with one probability, and check how the refusal threshold behaves on long inputs.
• Read training and test error together. Nearly equal errors point to underfitting or a data problem.
• Plot the worst confident mistakes. A shared cause among them points to the input pipeline.
• Widen the crop, then tune the model size within a computational budget.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 11.6 (pages 440–442), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.