Convolution and Pooling as an Infinitely Strong Prior

A rubber stamp as a strong prior

A rubber stamp has one pattern on its face. You press it at many places on a sheet, and the same pattern appears each time. The ink lands only where the face touches the paper. The stamp carries no separate design for each spot.

A convolutional layer works in the same way. The kernel is the pattern, the weights are the ink, and the same weights are pressed at every position of the input. This module asks what that repetition assumes, and when the assumption helps.

Section 9.4 frames the question with the idea of a prior. A prior is a probability distribution over the parameters that records what you believe before any data arrive. Combining the prior with the data gives the posterior, which is the belief after the data. Bayes’ rule gives the combination:

Here is the posterior, the likelihood of the data for each value of , and the prior. The symbol ∝ means that the constant which makes the posterior add up to 1 is left out. The book recalls priors from its Section 5.2. The rule itself is not numbered in Section 9.4.

• A weak prior has a high spread. The data can move the parameters more or less freely.
• A strong prior has a low spread. It plays an active part in deciding where the parameters end up.
• An infinitely strong prior gives zero probability to some parameter values. Those values are forbidden, whatever the data say.

The demo lets you feel the difference. The prior is a bell curve centered at 0 with spread . The data are readings with mean , and each reading has noise spread , fixed at 1. Each curve is scaled so that its highest point is 1, which lets the shapes be compared.

With large, the posterior follows the data. With small, it stays near 0 even when the readings average 2. At the prior is a spike, and the data cannot move the estimate at all. Switch the forbidden zone on and the posterior puts no probability at all above 0.5, however large the data mean. That is the book’s infinitely strong prior: the data choose only among the values the prior allows.

The forbidden zone is the book’s definition written as a condition on the prior. The line below restates it in symbols, with the set of forbidden values:

For a Gaussian prior and Gaussian noise the posterior has a closed form. The two equations below assume a prior mean of 0, as in the demo. This standard result is added here and is not printed in the book.

The posterior mean is the data mean shrunk towards 0, and the shrinking depends on . With , and , the posterior mean is 0.10, although the data average 2. A small spread allows the prior to outvote the data.

Locality and sharing as a prior on weights

Section 9.4 claims that a convolutional layer is a fully connected layer with an infinitely strong prior on its weights. The book states the prior in two parts. The first is locality: a weight is zero unless its input lies inside the small window of its output. The second is sharing: the weight that links an input to an output depends only on how far apart they are, so the weights at every position are shifted copies of one another.

A fully connected layer with inputs and outputs computes , with a full matrix of free weights. Every output reads every input. Under the two-part prior the matrix takes the form

Here is the window width and the kernel. The equation is added here as the algebra of the two parts, since the book states them in words. The nonzero entries sit in a band of width , and each diagonal of the band is constant. A matrix with constant diagonals is called a Toeplitz matrix. The layer therefore has free numbers, where a dense layer has .

The demo starts from a dense layer with 6 inputs and a random weight matrix. Switch on the window, then the sharing, and watch the free weights disappear. Select a cell and move the value slider. In the shared layer, the value you set is written onto every cell of its diagonal, and the readout says how many cells moved.

The shared layer is a convolution, and its outputs equal the sliding sum over its window. The readout checks the two routes against each other. Sharing is an exact constraint in the book’s sense, not a penalty. Changing one diagonal changes all of it and nothing else.

Forcing a dense layer onto the prior picks the closest member of the constrained family. For a least-squares distance, the closest convolution keeps the window and replaces each diagonal by its average. The formula is added here and is not printed in the book. The demo uses this average when you switch sharing on.

Pooling as invariance to small shifts

Section 9.3 introduced max pooling. Each output is the largest detector response inside a block of neighboring positions. The book reads pooling as a prior too: pooling is an infinitely strong prior that each unit should be invariant to small translations.

Write the detector responses as and the pooled outputs as . With pool width , the blocks are consecutive runs of positions, indexed by from 0:

The notation is added here. The operation is the one of Section 9.3. The invariance it gives is local. If a shift keeps every spike inside its block, the pooled output does not change. A shift that carries a spike across a block edge does change it. The demo uses one spike of height 1, a detector that fires at a single position.

Inside a block the pooled output records that a spike exists, but not where it sits. Across a block edge the output changes. A wider pool gives a larger invariant region, and it discards more information about position.

When the prior underfits

A prior helps only when its assumption holds. The book warns that convolution and pooling can cause underfitting for exactly this reason. If a task depends on precise spatial information, pooling every feature can raise the training error. Some architectures therefore pool some channels and not others, which keeps invariant features where they help. A task that combines information from very distant locations may also suit the local prior of convolution poorly.

The demo measures two things for a single spike placed, with equal chance, at each of the 16 positions. The presence test asks whether the pooled output shows that a spike occurred. The localization test asks how far the best guess lies from the true position, using the pooled output alone. Inside a block the best guess is the block center. For full blocks the root mean square error is

This formula is added here and is not printed in the book. It is the standard deviation of a uniform choice among the positions of a block, and it holds when the block width divides 16.

Presence stays perfect at every width, because a maximum keeps a spike whatever its position. Localization worsens as the block grows. The prior removes a distinction that this task needs, which is underfitting in the book’s sense.

Comparing models fairly

The book ends the section with a point about benchmarks. A convolutional model has the spatial relationships built in. A model without convolution would learn the same task after the pixels are relabeled, because it never assumed where the neighbors are. The book therefore argues that convolutional models should be compared with other convolutional models, and that models which must discover the layout should be compared with each other.

Relabeling the inputs is a permutation . The outputs do not change if the weights are relabeled in the same way:

This identity is added here. A dense layer is closed under relabeling, because is still an arbitrary matrix of free weights. A convolution is not closed. After a shuffle its nonzero weights no longer lie in a band. The demo shuffles the 16 inputs of a convolution with width 3 and counts how many of its 42 nonzero weights stay where the band put them.

A benchmark that compares a convolutional model with a dense model has to say which structure each one was given. The shuffled band is the structure that the convolution would have to rediscover from the data.

• Convolution is an infinitely strong prior on a layer: each output reads only its window, and every diagonal of the weight matrix is shared, so the layer has free numbers where a dense layer has .
• Pooling is an infinitely strong prior of invariance to small shifts. Max pooling keeps whether a feature is present in a block and discards its exact place inside the block.
• A prior helps only when its assumption holds. Pooling every feature can raise the training error when the task needs precise position, which is why some networks pool only some channels.
• Benchmarks should compare models with the same built-in structure. Relabeling the inputs leaves a dense layer dense, but it breaks the band of a convolution.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.4 (pages 345–347), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.