Data Types

The big picture

A convolutional network does not read only pictures. Its input is usually a set of channels, and each channel is the observation of a different quantity at some point in space or time. A microphone gives one channel of values over time. A color photo gives three channels over two axes of space.

The book gives examples of these data types in its table 9.1. The table varies two things: the number of axes that the kernel can slide along, and the number of channels at each point. Both are shown in the shape below, where is the number of channels and is the number of axes.

This shape notation is added here and is not printed in the book.

The readout of the demo below names the data type that matches the two settings. The names come from table 9.1 for the cases that the table lists. The demo is original, and it changes the axes and the channels with two sliders.

The kernel slides along every axis, and at each place it sums over all the input channels. The number of axes decides the shape of the sliding, and the number of channels decides how many values sit at each place.

Shifts in time and space

The book notes that convolution over a time axis makes the model equivariant to shifts in time, and convolution across a frequency axis makes it equivariant to frequency. Equivariance means that when the input moves, the output moves by the same amount. The book applies this to the spectrogram of a melody. The same melody in a different octave gives the same pattern at a different height.

The demo below uses one pulse and the kernel . The pulse moves by places. The output is the valid convolution, so it has values. This statement is added here and is not printed in the book:

The shift is exact only when the pulse stays away from the edges. A pulse that leaves the input would change the output in a way that no shift could explain. The valid convolution also makes the output one place shorter at each edge, so the peak of the output lies one place before the peak of the input.

Inputs of different sizes

The book points out that convolution can process inputs with a varying spatial extent. A collection of images can have a different width and height in each image, and the same kernel is applied a different number of times for each size. A fully connected layer cannot do this, because its weight matrix has a fixed size. The book adds that convolution can be written as matrix multiplication, and that the same kernel gives a different doubly block circulant matrix (a matrix built from shifted copies of the kernel) for each input size.

Sometimes the output may also vary in size, for example when every pixel needs a label. When the network must produce a fixed-size output, for example one label for the whole image, the book says that a pooling layer is added whose regions scale with the size of the input. Then the number of pooled outputs stays fixed. The demo below pools the convolution output into four regions.

The pooled output always has four values, whatever the input length. The regions grow with the input, so a longer input gives longer regions and not more outputs. This is the strategy that the book's figure 9.11 illustrates.

When convolution does not fit

Variable size is not enough for convolution to make sense. The book says that the input must vary because it holds more or fewer observations of the same kind of thing. Different lengths of recordings over time are fine, and different widths of observations over space are fine. If the slots of the input hold different kinds of observation, the same weights would be applied to unrelated quantities.

The book gives an example. In a college application, the features may include grades and standardized test scores, and not every applicant took the test. Convolving the same weights over the grades and the test scores does not make sense. The game below asks you to sort six cases.

In short, convolution needs the same kind of observation at each place along the axis. The axis can be a time axis, a space axis or a frequency axis, and the length of the input can vary. If each slot means something different, a layer that gives each slot its own weights, such as a fully connected layer, is the natural choice. This choice is added here and is not printed in the book.

What to take away

• The input has axes and channels. The table of data types in the book lists examples of each combination.
• Convolution over a time or frequency axis makes the model equivariant to shifts along that axis. When the input moves, the output moves by the same amount, away from the edges.
• Convolution handles inputs of varying length. A pooling layer with regions that scale with the input gives a fixed number of outputs.
• Convolution does not make sense when different slots hold different kinds of observation, because the same weights would read unrelated quantities.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.7 (pages 360–361), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.