Pooling

One summary per neighborhood

Picture a row of desks in a classroom, in groups of three. Each desk has a test score. The teacher needs one number per group for her report, so she writes down the highest score in each group. Her report is much shorter than the class list, and it still shows where the strong results are. Pooling does the same to a row of detector values: it replaces the values in a small region with one summary of them.

A convolutional layer has three stages, and pooling is the last one. The book sometimes calls the second stage the detector stage, and its Figure 9.7 shows the same layer in a simpler terminology where pooling is a layer of its own. Tap each stage.

Predict before you read on. The six cells below are detector values in two groups of three. Max pooling gives one value for each group.

Max pooling reports 7 for the first group and 5 for the second. The 2 and the 3, and the 1 and the 4, do not reach the output at all. Section 3 shows what that buys.

Pooling functions

A pooling function takes the values in one region and returns one number. The book gives max pooling as the example, and names three other popular choices: the average over a rectangular neighborhood, the L2 norm over it, and a weighted average whose weights depend on the distance from the center. For a width of and a stride of , the region covers the positions , and . In general, in notation added here, since the book gives these functions in words:

The four functions differ in what they read from the region. Max pooling reads only the largest value.

Average pooling uses every value equally. The L2 norm is larger for regions with several strong values.

The distance-weighted average gives the center the most weight. The weights 0.25, 0.5 and 0.25 used here are an example chosen for this module, and the book does not print them.

Lower a small value inside a window and watch the max. It does not move, because the largest value is still the largest. The average moves at once. This is the first clue to the invariance that Section 3 explains.

Invariance to small shifts

Move an input by one place and most detector values change. Pooling is meant to make the summary change less. The book says that pooling makes the representation approximately invariant to small translations. Here invariance means that the pooled output stays the same when the input moves a little. Figure 9.8 shows max pooling with a width of three and a stride of one.

Call the shifted row , in notation added here. Each value moves one place to the right, and a zero enters at the left, so :

The pooled output of the shifted row is . The figure compares the two rows at the same position , and so does the demo. An output is unchanged when the two values match:

In the default row, max pooling keeps 4 of 8 outputs in place after a shift of 1, and average pooling keeps none. Figure 9.8 changes half of the four outputs it shows, so the figure and the demo describe the same effect. A single peak away from the ends of a row of zeros is the easiest case: its shift changes only the two outputs whose windows gain or lose the peak.

When the question is whether an eye sits on the left of a face, rather than exactly where, invariance helps. When the question is whether two edges meet at a precise corner, location matters, and the layer must keep it. The book describes pooling as an infinitely strong prior that the layer must be invariant to small translations, which helps greatly when that assumption is correct.

Learned invariances

Figure 9.9 goes one step further. Three separate filters each respond to a slightly different angle of the same handwritten 5. A max over their outputs gives a large response whichever filter matches. The pooling unit therefore responds in about the same way to both inputs, although the filters that fire differ. The book adds that pooling over separately parametrized convolutions lets the features learn which transformations to become invariant to.

The demo is a one-dimensional toy of this idea, not the book's experiment. A stimulus has an angle . Each filter responds most strongly at its preferred angle , and its response falls with the distance from that angle. The width of the fall is . The formula is added here, since the book prints none:

The pooled unit takes the largest of the filter responses:

Downsampling with a stride

Pooling does not have to visit every position. The book describes pooling regions that are spaced pixels apart instead of one pixel apart. The stride is the distance from the start of one region to the start of the next. Figure 9.10 uses a width of three and a stride of two on six detector values. Its outputs are 1, 0.2 and 0.1. The last region covers only two values. The book says it must be included, or the detector values it covers are ignored.

Fewer pooled values mean less work for the next layer. The book's example halves the representation size, which reduces the computational and statistical burden on the next layer. For a general stride the next layer has about times fewer inputs. With detector values, the number of pooled outputs is:

If only full regions are kept, the count becomes:

For the six values of Figure 9.10, the first formula gives three outputs and the second gives two. The partial region at the edge makes the difference. Both formulas are added here and are not printed in the book.

Summaries for inputs of any length

A classifier needs a fixed number of inputs, but images come in different sizes. The book's answer is to pool so that the layer always passes the same number of summaries on. Its example is four summaries, one for each quadrant of an image, whatever the image size. The book does this by varying the size of the offset between pooling regions, so the regions stretch or shrink with the input.

This module uses one concrete rule of that kind. The bins start and end at fixed fractions of the input length, so each bin holds about the same number of values. This rule is added here and is not printed in the book.

Bin holds the detector values from up to , and its pooled output is their maximum:

Fixed windows behave differently. Width three and stride one gives outputs, so the output grows with the input. Bins fix the output size instead, and the rest of the network can then be a fixed-size classifier.

• A pooling function replaces a region with one summary. Max pooling reads only the largest value, so a small shift changes few of its outputs, and a change to a smaller value leaves it unchanged.
• A stride spaces the regions out and shrinks the representation by about . The partial region at the edge must be kept.
• Pooling into a fixed number of bins gives a fixed-size output for an input of any length.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.3 (pages 339–344), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.