Computer Vision

Computer vision and the scale of pixels

Vision is easy for people and many animals, and hard for computers. That is why computer vision has been one of the most active areas for deep learning. Most of the standard benchmarks are object recognition or reading characters, and most of the work aims at human abilities.

Most deep learning for vision names the object in an image, draws a box around it, transcribes a sequence of symbols, or labels every pixel with its object. Image synthesis matters mainly for image restoration, such as repairing defects or removing objects from a picture.

Before a network sees an image, its pixels should lie in one reasonable range, such as or . Predict first.

The unit is a toy, and it is added here. It adds up sixteen pixels, each with weight 0.075, and passes the sum through tanh. The formula is below. The book does not print it.

Changing the scale multiplies every pixel by 255, so the weighted sum is 255 times larger. The tanh then saturates near 1, and the unit stops telling the patches apart. The book’s advice is to put all images on one scale before training.

Scale is the only preprocessing the book calls strictly necessary. Many architectures also need each image cropped or scaled to one standard size, although some convolutional models accept any size and resize their pooling regions to match. Other preprocessing removes variation that a designer can describe and is confident has no bearing on the task. With large datasets and large models that step is often unnecessary, and it is better to let the model learn which variations to ignore. The book notes that AlexNet subtracts only the mean of each pixel across the training examples.

Global contrast normalization

Preprocessing removes variation that a designer can describe and is sure has no bearing on the task. Contrast is one of the most obvious such variations. In deep learning, the contrast of an image usually means the standard deviation of its pixels, their spread around the mean. Global contrast normalization (GCN) subtracts the mean of the image and rescales it so that the spread equals a constant .

The demo changes the brightness, the contrast amplitude and the scale. The formulas are below. The mean in and the contrast in are both computed over the whole image. Equation (12.1) defines the contrast from the mean, and (12.2) defines the mean.

The denominator of equation (12.3) is the contrast from (12.1) with the regularizer added inside the root. The book says that dividing by a true standard deviation of a nearly flat image only amplifies sensor noise. Adding keeps that from happening, and keeps the division safe when the contrast is zero.

Brightness has no effect on the output, because the mean is removed first. Equation (12.3) divides by the contrast, so scaling the image by a constant also has no effect when is zero. The book notes that Coates et al. (2011) used = 0 and = 10 on small patches, where nearly constant intensity is common.

The book gives three practical choices. Large images cropped around objects are unlikely to be nearly flat, so = 0 and = 10⁻⁸ are safe, as used on CIFAR-10. Small patches cropped at random are often nearly flat, so a large regularizer helps more. The scale can usually be 1. The standard deviation in (12.3) is a rescaled L2 norm of the mean-removed image, and it is preferred because its division by the number of pixels lets one serve images of any size.

Mapping examples onto a sphere

The book’s Figure 12.1 shows GCN as a map onto a sphere. The figure is a picture of many dimensions, so the demo below uses the smallest case where the formula can be checked by hand: two numbers. This is a rebuild of the idea and not a copy of the figure. With two numbers the mean-removed pair lies on one line, so the sphere collapses to two points.

For two numbers the book’s formula becomes the two lines below. This form is added here, and it is equation (12.3) for two inputs, with the book’s , and .

With at zero, every point off the diagonal goes to one of two corners, and the output is in size. With above zero, a point whose two numbers are close is pulled toward zero. That is the book’s point about regularized GCN: it draws examples toward the sphere but does not discard their size completely.

The book gives a reason for the sphere. Neural networks respond better to directions in space than to exact locations. A network that must respond to several distances along one direction needs hidden units with collinear weights and different biases, which is hard to learn. GCN reduces each example to a direction. The book also warns that sphering, which rescales the principal components to equal variance and is also called whitening, is a different operation from GCN.

Local contrast normalization

Global normalization can hide edges and corners. Imagine a city square where half the image is in shadow. GCN makes the difference between the two halves large, but it does not make the edges inside the dark half stand out. The book’s Figure 12.2 shows this with real images.

Local contrast normalization (LCN) repeats the same idea in small windows. Each pixel has the mean of its neighbors subtracted and is divided by their standard deviation. The book notes that a window that is too wide loses fine texture. The demo below uses a row of values instead of an image, so that the effect can be read on one line.

The formula below is added here. The book describes LCN in words, and this is the one-dimensional version of that description. The window at position holds the positions within of it, so it has values. The spread is the standard deviation of those values. At the two ends of the row the window is cut short, and the mean and the spread are taken over the values that exist.

A narrow window sees only the local texture, so its standard deviation is small and the texture is strongly amplified. Regularization with stops that amplification where the window is almost flat. The book says the same about small windows: they are more likely to be nearly constant, so regularization matters more.

The book adds three practical points. LCN can be computed efficiently with separable convolutions that give local means and local standard deviations, followed by element-wise subtraction and division. It is differentiable, so it can also act as a nonlinearity on hidden layers, not only as preprocessing. For color images, some methods normalize each channel separately, while others combine the channels to normalize each pixel.

Dataset augmentation and voting

Dataset augmentation adds copies of the training examples that have been changed in ways that keep their class. The book says object recognition suits this well, because the class survives many transformations. The book calls augmentation an excellent way to reduce generalization error for most vision models. Its examples include random translations, rotations and, in some cases, flips, along with color changes and nonlinear distortions.

A related idea works at test time. The book shows the model many versions of the same input, such as crops at slightly different positions, and lets the copies vote. That is an ensemble, and the book says it helps reduce generalization error. The demo below shifts a letter-shaped object to show why a small shift keeps the class.

The model learns from the shifted copy as if it were a new picture of the same letter. This is the invariance the book describes. A flip is a different matter. The mirror image of an L is not an L, so a flip is only a valid copy where the class survives it, which is why the book allows flips only in some cases. Shifts that are too large can push the object out of the frame, and then the copy no longer shows the whole class.

• All images should share one pixel scale, because a network sums its pixels with weights. Mixing scales usually fails, as the book warns.
• GCN removes the mean and fixes the spread, so brightness and, with the regularizer at zero, overall scale drop out of the output.
• The regularizer keeps GCN and LCN from amplifying noise where the contrast is nearly zero.
• LCN normalizes small windows, so it shows edges and texture that a global normalization averages away.
• Augmentation copies examples under changes that keep the class. Voting over test-time copies is an ensemble that reduces generalization error.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 12.2 (pages 452–458), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.