The Neuroscientific Basis for Convolutional Networks

A cartoon of the visual cortex

Think of a row of tuning forks. Each fork rings loudly for one pitch and stays almost silent for the rest. Some neurons in the early visual system behave the same way for orientation. They answer strongly to a bar at one angle and hardly at all to bars at other angles.

The book describes the experiments of Hubel and Wiesel, who recorded the activity of individual neurons in cats while patterns of light were projected in front of them (Hubel and Wiesel, 1959, 1962, 1968). Their most influential finding was selectivity: neurons in the early visual system responded most strongly to very specific patterns, such as precisely oriented bars. Their work was later recognized with a Nobel prize.

The book focuses on V1, the primary visual cortex, which is the first area of the brain to process visual input in a significant way. Light reaches the retina, passes through the optic nerve and the lateral geniculate nucleus, and arrives at V1. The book treats those two steps as a relay that carries the signal.

A neuron’s receptive field is the small region of the image where a change in light alters its response. The demo below uses a simulated neuron whose response depends only on the angle of a bar placed in its receptive field.

The response is a single bump. It is close to its maximum near the preferred angle and falls as the bar turns away. This curve is a toy. The Gaussian shape, its width of 20 degrees and its center at 60 degrees were chosen for this page. Orientation repeats every 180 degrees, so the curve wraps around: a bar at 0 degrees and a bar at 180 degrees are the same bar. The book does not give a tuning curve for a real cell.

Simple cells, complex cells and small shifts

The book names three properties of V1 that convolutional layers are designed to capture:

1. A spatial map. V1 has a two-dimensional layout that mirrors the image on the retina. Light in the lower half of the retina affects only the matching half of V1. Convolutional networks keep this by defining their features on two-dimensional maps.
2. Simple cells. The activity of a simple cell is, to some extent, a linear function of the image inside a small, localized receptive field. The detector units of a convolutional network are designed to emulate this.
3. Complex cells. A complex cell responds to features like those of simple cells, but it is invariant to small shifts of the feature. This inspired the pooling units of convolutional networks. The book adds that complex cells tolerate some changes in lighting too, which inspired cross-channel pooling such as maxout units.

Invariant to small shifts means that the response stays much the same when the feature moves a little inside the receptive field. The demo below builds a simple cell that reads a window of four cells, and a complex cell that keeps the largest response of the simple cells next to it.

With the pooling radius at 0, the complex cell is the same as the simple cell. Each step of the radius adds one neighbor on each side, and the range of bar positions that give a response grows by two. The response of the complex cell therefore stays up over a wider stretch of the row. That stretch is the invariance the book describes.

Stacking detection and pooling

The book says that stacks of layers that alternate between selectivity and invariance can yield grandmother cells, units that respond to one concept wherever it sits in the input. The book cites Quiroga et al. (2005) for neurons in the human medial temporal lobe that respond to specific people. One such neuron fired for photos of a particular actress, for her drawings and even for the written name. The book notes that these neurons are more general than convolutional units, which would not generalize from a name to a face.

The closest analog of the last layer of features is the inferotemporal cortex (IT). Information flows from the retina through the LGN to V1, then V2, V4 and IT, within the first 100 milliseconds of a glimpse. The book reports that convolutional networks can predict IT firing rates, and perform similarly to time-limited humans on object recognition (DiCarlo, 2013).

The demo below stacks two detect-and-pool stages above a row of inputs. Each pooling stage widens the set of positions at which the top unit responds.

Without pooling, the top unit responds only when the bump sits at its own location. Each pooling stage adds one position on each side. Two stages give five positions, and the top unit then responds to the same pattern at several places. A real grandmother cell is far more tolerant than this toy, but the principle is the same: selectivity from detection, invariance from pooling.

Where the cartoon breaks

The book lists ways in which the picture above differs from the real visual system. Some differences are known to computational neuroscientists and lie outside the book. Others are not yet known, because many basic questions about the visual system are unanswered. In the book’s order, the main ones are these.

• Resolution. The human eye is mostly low resolution, except for a tiny patch called the fovea. The fovea sees an area about the size of a thumbnail held at arm’s length. We feel that we see a whole scene sharply because the brain stitches together several glimpses. Most convolutional networks receive large full-resolution photographs. The brain makes quick eye movements, called saccades, to look at the most salient parts. Adding similar attention mechanisms to deep learning models is an active research direction.
• Other senses and moods. The visual system is integrated with hearing and with factors such as mood and thought. Convolutional networks so far are purely visual.
• Scenes and three dimensions. The visual system understands whole scenes, with many objects and the relations between them, and it processes rich three-dimensional geometry. Convolutional networks have been applied to some of these problems, but those applications are in their infancy.
• Feedback. Even V1 is heavily affected by feedback from higher levels. Feedback has been explored in network models, but it has not yet been shown to offer a compelling improvement.
• The units themselves. The brain probably uses different activation and pooling functions. A neuron may not be well described by one linear filter. A recent model of V1 uses several quadratic filters for each neuron (Rust et al., 2005). The book also suggests that simple and complex cells may be one kind of cell, whose parameters give a continuum of behaviors.

The book adds that neuroscience has told us relatively little about how to train convolutional networks. The early training milestones are in the timeline at the end of this section.

The demo below makes the first difference concrete. The sharp circle is a teaching device for the fovea, not a model of its shape. The picture is blurred everywhere except inside the circle.

The book places four early milestones on pages 367 and 368. Each one answers a question about how convolutional networks were trained: first with parameter sharing and no back-propagation, then with back-propagation through time-delay networks, and finally on images. Press each year to read its milestone.

Gabor functions and reverse correlation

How does a neuroscientist find the weights of a simple cell? In a biological network we cannot read the weights directly. Instead we put an electrode in the neuron, show several samples of white-noise images in front of the animal’s retina, record how each sample changes the response, and fit a linear model to the responses. This approach is called reverse correlation (Ringach and Shapley, 2004). In an artificial network we only need to display the convolution kernel.

Reverse correlation shows that most V1 cells have weights described by Gabor functions. Treat the image as a function of two coordinates, . A simple cell samples the image at a set of x coordinates and y coordinates , applying weights that depend on the location. Its response is

The weights take the form of a Gabor function with the parameters listed in equation 9.16, where the two coordinates are first shifted and rotated by equations 9.17 and 9.18:

• The coordinate system. and give the center of the receptive field, and rotates the axes. The cell responds to features centered at , and to changes in brightness along a direction rotated radians from the horizontal. That direction is the axis, and the stripes lie across it, at radians.
• The Gaussian factor. The factor is a gate that keeps the response near the center. The scale sets the size of the response, and and set how quickly it falls off.
• The cosine factor. The factor controls how the cell responds to changing brightness along . is the frequency and is the phase offset.

The cell is most excited when the image is bright where its weights are positive and dark where they are negative. It is most inhibited when the brightness wave is fully out of phase with its weights. The book’s Figure 9.18 shows Gabor functions with three sets of parameter changes: the coordinate system, the Gaussian widths, and the frequency and phase. The demo below rebuilds that idea with one Gabor function that you control.

Two settings do most of the work. The orientation decides the direction in which brightness changes, and the stripes lie across that direction. The width parameters decide how many stripes are visible inside the envelope. The demo sets to 1, and it puts the center at the origin, so and are both 0.

Recovering a hidden weight pattern

Reverse correlation can be tested on a simulated unit. The unit below is linear, and its weights are a hidden Gabor pattern. It responds to each white-noise image with = the weighted sum of the image pixels, plus a little noise . The demo estimates the weights from the recorded responses only. The estimate is found by a ridge-regularized least-squares fit. This estimator is added here and is not printed in the book:

Here numbers the samples, and is how many have been recorded. The small constant keeps the solution stable when there are few samples. The demo’s noise has a fixed size, so the error shrinks as the number of samples grows, but it does not reach zero.

Ten recorded responses are not enough to fix eighty-one weights, so the estimate is weak. A hundred responses already give a clear picture, and several hundred give a close match. The correlation in the readout is a score that does not depend on the overall scale of the weights.

Quadrature pairs and the complex cell

The book describes a complex cell as computing the L² norm of the responses of two simple cells, as in the equation below. This equation is printed in the text of the book and is not numbered there.

An important special case occurs when the second simple cell, , has the same parameters as the first, , except for the phase , which is set so that is one quarter cycle out of phase with . In that case the two cells form a quadrature pair. A complex cell built this way responds when a high-amplitude sinusoidal wave of the right frequency and direction appears near the center of its field, regardless of the phase of that wave. It is invariant to small translations of the image along its direction, and to negating the image.

The reason is a short calculation. (This derivation is added here and is not printed in the book.) Take a grating of amplitude and phase . The even filter responds in proportion to the cosine of the phase, and the odd filter responds in proportion to the sine, with the same constant for both:

When the two responses are squared and added, the phase drops out, so the complex cell gives a constant:

The two simple cells change with the phase, but the complex cell does not. The demo below checks this with a Gaussian-windowed even filter and odd filter applied to a sine grating whose phase you move.

The even cell and the odd cell trade their responses as the phase turns, and their squares always sum to the same value. Negating the image shifts the phase by half a cycle, so the complex cell is unchanged by negation as well. These are the invariances the book describes.

Learned features that look like Gabor functions

Olshausen and Field (1996) showed that a simple unsupervised algorithm, sparse coding, learns features with receptive fields similar to those of simple cells. The book reports that a wide variety of statistical learning algorithms since then learn Gabor-like features when they are applied to natural images, and that this includes most deep learning algorithms in their first layer.

The book is careful about what this shows. Because so many different algorithms learn edge detectors, learning an edge detector does not single out one algorithm as the right model of the brain. The book also notes that an algorithm that fails to learn some kind of edge detector on natural images is a bad sign.

Figure 9.19 shows two examples. The left panel holds weights learned by spike-and-slab sparse coding on small image patches. The right panel holds first-layer kernels of a fully supervised convolutional maxout network. The demo below does not reproduce either panel. Its target is a made-up kernel that looks like a learned edge detector, built from Gabor parameters that you must recover by matching it. It is not a kernel from the book.

The similarity score is the correlation between your Gabor function and the target, over every pixel. A score of 1 means the same shape up to brightness and contrast. Each slider has to land on the right value. A one-degree error in the orientation or the phase, or one step in the frequency or the width, drops the similarity below the target.

• V1 neurons are selective for orientation and location. Simple cells respond linearly inside a small receptive field, and complex cells tolerate small shifts.
• Pooling over neighboring simple cells widens the range of positions over which a feature is detected. Stacking detection and pooling can yield grandmother cells.
• Gabor functions describe the weights of most V1 simple cells. Their orientation, frequency, width and phase are the parameters that matter, and reverse correlation estimates them from responses to white noise.
• A quadrature pair of simple cells gives a complex cell whose response does not depend on phase.
• Many learning algorithms find Gabor-like edge detectors on natural images. That does not single out one algorithm as the right model of the brain.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.10 (pages 364–371), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.