Convolutional Networks and the History of Deep Learning

A short history

Convolutional networks sit at the center of the history of deep learning. The book calls them a key example of a successful application of insights from the brain to machine learning. They were also among the first deep models to perform well, long before deep models in general were considered viable.

The timeline below lists the milestones that the book names on pages 371 and 372, from the 1990s on, in the book's order. The earlier milestones from 1976 to 1989 are in the previous module, on section 9.10. Press each year to read what happened.

A convolutional network developed by the neural network research group at AT&T was used to read checks. The book adds one number that shows how far this reached.

The book says that by the end of the 1990s this system, deployed by NEC, was reading over 10% of all the checks in the US. The word over matters. The book gives a lower bound and no exact figure, so any guess above 10% fits the text, and a guess of 1% does not.

Convolutional networks were also used to win many contests. The current intensity of commercial interest in deep learning began when Krizhevsky et al. (2012) won the ImageNet object recognition challenge, although convolutional networks had won other contests for years earlier with less impact. Microsoft later deployed optical character recognition and handwriting systems based on convolutional networks (Simard et al., 2003). Chapter 12 covers these applications in more detail.

Why convolutional networks worked first

The book notes that convolutional networks were among the first working deep networks trained with back-propagation. It says that it is not entirely clear why they succeeded when general back-propagation networks were considered to have failed, and it offers three possible reasons:

1. Convolutional networks were more computationally efficient than fully connected networks, so it was easier to run many experiments and tune the implementation and the hyperparameters.
2. Larger networks also seem to be easier to train.
3. The primary barriers may have been psychological. Practitioners did not expect neural networks to work, so they did not make a serious effort to use them.

The book adds that with modern hardware, large fully connected networks appear to perform reasonably on many tasks, even with datasets and activation functions that were available and popular when fully connected networks were believed not to work well. The demo below puts a number on the first reason. It compares how many parameters a dense layer and a convolutional layer need for the same grayscale image with one input channel. The formulas are added here and are not printed in the book.

The dense count uses one weight for every pair of input and output units, and one bias for each output unit. The convolutional count uses one small kernel for each output channel, and that kernel is reused at every position in the image. This sharing is why the convolutional count stays small as the image grows.

Stacking layers grows the receptive field

The book says that convolutional networks provide a way to scale models to very large size on data with a clear grid-structured topology. One reason this works is the receptive field: the region of the input that one unit of a deep layer can see. A unit in the first layer sees a small patch. A unit in a later layer sees a larger patch, because it reads from units that already saw patches of their own.

Two numbers describe each layer. The field is the number of input pixels that a unit in layer can see, and is the distance between neighboring units in that layer, measured in input pixels. The recurrence below, with as the stride of the pooling after each layer, is added here and is not printed in the book.

Both start at 1 before the first layer. A convolution with kernel side widens the field by steps, each step being one gap between neighboring units. Pooling with stride multiplies the gap, so later layers widen the field by more. The recurrence assumes that the pool keeps every p-th unit and reads no window of its own.

Without pooling, the field grows by the kernel's reach at every layer and stays modest. A stride of 2 after each layer doubles the spacing at every step, so the same number of layers reads a much wider stretch of the input. This is the same idea that the stacked detect-and-pool stages of Section 3 of the previous module used.

Grid-structured data and the handoff to Chapter 10

The book says that convolutional networks provide a way to specialize neural networks to work with data that has a clear grid-structured topology. This approach has been most successful on a two-dimensional image topology. One-dimensional sequential data is handled by a different specialization, recurrent neural networks.

The sort game below asks you to place eight kinds of data into three bins: grid-structured data, sequences, and neither. Press a bin for each item. The page tells you whether each choice is right, and it counts your correct choices. The bins grid and sequence follow the book's split. The bin neither, and the eight examples, are added here and are not printed in the book.

Chapter 10 picks up the sequence case. Recurrent networks handle one-dimensional data read in order, such as audio, text and prices. This is a preview, and the next chapter covers it in full.

• Convolutional networks were among the first deep models to perform well, and among the first trained with back-propagation.
• Early commercial uses included check reading (NEC, reading over 10% of US checks by the end of the 1990s), OCR systems at Microsoft, and contest wins such as ImageNet in 2012.
• A convolutional layer needs far fewer parameters than a dense layer on the same image, because one small kernel is reused at every position.
• Stacked layers grow the receptive field, and pooling with a stride above 1 makes that growth much faster.
• Convolution suits grid-structured data, especially images. Sequential data is the subject of Chapter 10.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.11 (pages 371–372), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.