Variants of the Basic Convolution Function

The big picture

Imagine a rubber stamp with a fixed pattern on it. You press it at the first place on a strip of paper, lift it, slide it one place to the right, and press again. Every press leaves an inked mark whose strength says how well the pattern fits the paper at that place. A convolutional layer works like this. The pattern is a small array of weights called the kernel , and the same kernel is applied at every place of the input.

The book describes the variants of this idea that networks actually use. Each variant changes one thing about the stamp: how many stamps there are, how far it moves, how the edge of the paper is treated, which weights exist at each place, and how training passes errors back through it. Here is the key row used in every equation of this page.

The demo below uses one input signal and one kernel. A single kernel of width has three numbers, and the output at each place is the sum of those three numbers times the three inputs under the stamp. The signal is one-dimensional, so the stamp slides along one line. The same idea works with images and two dimensions, as the next sections show.

The output is one number for each place where the window fits, so it has values for an input of length . The rule is the same at every place: the window multiplies its three cells by the three kernel weights and adds the results. Nothing changes from place to place except the cells the window covers. That sharing is why a convolutional layer needs far fewer weights than a layer in which every output reads every input.

Many kernels for many channels

A single kernel finds one kind of pattern, but it finds it at every place. A layer needs many kinds of pattern at many places. So the book's convolution is really a bundle of convolutions, one kernel for each kind of feature, all applied in parallel.

The input is also more than a line of numbers. A color image has a red, a green and a blue value at each pixel, and the second layer of a network receives one channel for each kernel of the first layer. So the input is a stack of channels. The book writes the input as a three-dimensional tensor, with one index for the channel and two for the position, and it drops the batch axis that software adds. In this section the position is one-dimensional, so the input has two indices: channel and place.

Here is the book's formula for the multi-channel case, written with the book's own indices. The kernel has one entry for every output channel , every input channel , and every offset and inside the kernel. The output at channel and position , sums over all input channels and all offsets.

Read the equation from the inside out. The product is the input value that the kernel entry reaches, and the kernel entry says how strongly it counts for output channel . The sum over , and covers every input channel and every offset. The appears because the book counts positions from 1, as linear algebra does. Programming languages count from 0, which removes the .

The book also remarks that multi-channel convolution is not commutative unless the number of output channels equals the number of input channels. The demo lets you build one output channel from two input channels by hand.

Each output channel has its own set of kernels, one per input channel. The demo's output channel has weights: two input channels, each with three taps. Adding a second output channel adds another six weights, so the number of weights grows with the number of channels on both sides.

Stride: skipping output positions

Sometimes the output does not need every position. A stride tells the kernel how many input places to move between two outputs. The book writes this as a downsampled convolution. It keeps every -th output of the full convolution, and it reads the input at the places that the kernel reaches from each kept position.

The function takes the kernel , the input and the stride , and returns the output . The sum has the same form as equation 9.7, but the input position is now instead of . Each step of the output moves the window by places, and a stride of 1 gives back equation 9.7.

The book states that a convolution with stride 2 gives the same numbers as a convolution with stride 1 followed by keeping every second value (its figure 9.12). The two routes agree, but the second one computes values that are then thrown away. The demo compares the two routes and counts the multiplications.

With the kernel width and an input of length , a stride of 1 keeps all 10 positions. This derivation is added here and is not printed in the book: the number of kept outputs is . Check it with the demo at stride 2, which keeps 5 outputs, and at stride 4, which keeps 3.

Zero padding and the size of the output

Without any change, every layer makes the input narrower. A kernel of width cannot be placed past the last input, so an input of length gives outputs. Each layer loses places. Adding zeros around the input is the standard fix, and it lets the kernel width and the output size be chosen separately. The total number of added zeros is written .

The book names three special cases. With no zeros, the kernel only visits places where it fits completely. MATLAB calls this valid convolution, and the output is wide. With the zeros that keep the width, which the book calls same convolution, the output has the same width as the input. With enough zeros that every input place is visited times, the book calls it full convolution, and the output is wide.

Zeros have a cost. An input place near the border is covered by fewer windows than a place in the center under same convolution, and an output near the border reads fewer real inputs under full convolution. The book notes that the best amount of padding for test accuracy usually lies between valid and same. The demo below shows the width and the padded row.

The width is . This formula is added here and is not printed in the book, but it gives the three special cases above: for valid, for same and for full.

The bars show, for each output, how many of its three kernel cells fall on real input rather than on a zero. Under valid convolution every bar is 3. Under same convolution the two outputs at the ends see only 2 real inputs. The edge inputs also feed fewer outputs, which is the book's point about underrepresented borders.

The book's figure 9.13 uses an input of 16 places and a kernel of width 6. Without zeros the width falls to 11, 6 and then 1, and the third layer is the last one that a kernel of that width can cover. With five zeros in total at each layer the width stays at 16, so the network can be made as deep as the hardware allows.

Locally connected layers and channel groups

Sometimes the same pattern should not be reused everywhere. A locally connected layer keeps the small receptive field of a convolution, but gives every output position its own weights. The book writes this as equation 9.9, with a six-index tensor in which every output position has its own copy of the kernel.

The sum has the form of equation 9.7, with in place of . The only change is the index and inside , which selects a different set of weights at each output position. The book calls this unshared convolution. It is useful when a feature depends on a small region of the input, but the same feature has no reason to occur everywhere. For example, a detector for a mouth only needs to look in the lower half of a face image.

Figure 9.14 of the book compares three connection patterns on the same one-dimensional input. The locally connected layer and the convolution have the same connections and differ only in whether the weights are shared. The fully connected layer has no restriction on its connections at all. The demo below counts the weights of each pattern. This count is added here and is not printed in the book.

In the demo, the number of outputs is . A locally connected layer with one input channel and one output channel stores weights. A convolution stores only , and a fully connected layer stores . The gap grows quickly with the input length.

Channels can be restricted in a similar way. The book suggests letting each output channel depend on only some of the input channels, for instance by connecting the first few output channels to the first few input channels, and the later output channels to the later input channels. Figure 9.15 shows the pattern with two groups. The number of parameters falls, and the book notes that the number of hidden units does not change.

Tiled convolution

Tiled convolution is a compromise between a convolution and a locally connected layer. Instead of one kernel or one kernel per position, it learns a small set of kernels and rotates through them as the window moves. Neighboring positions use different kernels, as in a locally connected layer, but the number of stored weights grows only with the size of the set, not with the size of the output.

The output positions cycle through different kernel choices in each direction. The index is taken modulo , so the kernel for position has the index , counted from 1 as in the book. The book notes that if equals the output width, this is the same as a locally connected layer. The book also says the tiling range can differ in each direction.

The demo uses one-dimensional outputs. Each output position gets a letter for the kernel it uses, and the letters repeat after positions. This is the pattern of the tiled panel of figure 9.16 in the book.

The book notes an interaction with max pooling for locally connected and tiled layers. If their different filters learn different transformed versions of the same feature, the max-pooled units become invariant to that transformation (the book cites its figure 9.9). Ordinary convolution is hard-coded to be invariant to translation only.

Gradients and the transpose

Training needs the gradient of the cost with respect to every weight, and it needs the gradient with respect to the input so that the error can be passed to the layer below. The book shows that three operations are enough for a feedforward convolutional network: the convolution itself, the gradient with respect to the kernel, and the gradient with respect to the input. Three operations are also enough for the reconstruction functions used by autoencoders.

The convolution is a linear map, so it can be written as a matrix product after the input is flattened into a vector. The matrix is sparse, and each kernel entry is copied to several places in it. Multiplying by the transpose of that matrix is the operation that sends the output gradient back to the input. The book gives the two-dimensional, single-example version of the three operations. The demo below uses one dimension, which keeps the same structure.

Let be the gradient of the cost with respect to the output, so is . The kernel gradient is equation 9.11. It sums times the input that each kernel entry touched.

The gradient with respect to the input is in equations 9.12 and 9.13. Read equation 9.13 as a rule: an input place receives gradient from every output place and kernel entry whose window covers it. The constraints under the sums say exactly which windows those are. The stride decides how many windows cover each input.

The book warns that the input gradient needs care with the stride. Several input sizes can give the same output size, so the transpose must be told the original input size. The reconstruction in equation 9.14 uses the same function, applied to hidden units with kernel .

To train an autoencoder with this reconstruction, the decoder needs the gradient with respect to the kernel, which the book writes as . The encoder needs the gradient with respect to the hidden units, which the book writes as . The gradient with respect to the reconstruction is the input to both.

The demo uses a fixed input of nine places and a fixed target for the output. The cost is one half the squared difference between the output and the target, so the output gradient is the output minus the target. The readout checks equations 9.11 and 9.13 against a numerical derivative. The largest difference is at the level of rounding, which confirms the formulas.

Bias terms

A convolutional layer usually adds a bias to each output before the nonlinearity. The question is how many biases there are. The book gives four choices. A locally connected layer gives each unit its own bias. A tiled layer shares biases in the same tiling pattern as its kernels. A convolution usually has one bias per output channel, shared by every position in that channel map. When the input size is fixed, a separate bias for each output position is also possible.

The shared bias is the usual choice. A separate bias at each place costs a little statistical efficiency, but it lets the model correct for differences that depend on position. The book's example is implicit zero padding. Detector units at the edge of the image receive less total input, and they may need larger biases.

In the demo the input is a row of eight ones, the kernel has three taps that all equal 1, and the padding is same. Interior outputs sum three ones and the edge outputs sum only two, so the layer without biases gives 2, 3, 3, 3, 3, 3, 3, 2. A shared bias moves every output together. Separate biases can fix the edges without touching the interior.

What to take away

• A convolution applies one kernel at every place, and a layer runs many kernels in parallel, one for each output channel and input channel pair.
• The stride skips output positions. Computing every position and then discarding some gives the same numbers at a higher cost.
• Zero padding controls the output width. The width is , and keeps the width at every layer.
• Locally connected layers give each position its own weights, and tiled layers rotate through a small set of kernels. Both sit between a convolution and a fully connected layer.
• Training uses the kernel gradient of equation 9.11 and the input gradient of equations 9.12 and 9.13. The transpose of the convolution computes the input gradient.
• Biases are usually shared across each channel, and a separate bias per position can correct the edges that padding creates.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.5 (pages 347–357), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.