Structured Outputs

The big picture

A classifier answers one question about a whole image: is there a cat in it? A pixel labeler answers a question about every pixel: which class does this pixel belong to? The answer is a map, and the map has the same two position axes as the image.

The book writes the output as a tensor , where is the probability that pixel of the input belongs to class . Each value is one class at one pixel. The output therefore has a class axis and two position axes. Typically this object is just a tensor, emitted by an ordinary convolutional layer. It lets the model label every pixel and draw precise masks that follow the outlines of individual objects.

The demo below lets you paint the object pixels of an image by hand. The class count sets how many values the output tensor holds.

The number of values in is the number of classes times the number of pixels. For an image with four classes that is . This count is added here and is not printed in the book. The mask you paint is one class. The network produces one such map per class.

The output plane can be too small

Classification networks shrink the image on the way down, and the largest cut usually comes from pooling with a large stride. That is fine for one label per image. A pixel labeler needs a plane of similar size to the input, or its labels cannot line up with the pixels.

The book names three ways out. You can avoid pooling altogether (Jain et al., 2007), emit a lower-resolution grid of labels (Pinheiro and Collobert, 2014, 2015), or use a pooling operator with unit stride. The demo below pools cells. The window has width and moves cells each time.

The formula above is added here and is not printed in the book. It counts the windows that fit inside the input.

With unit stride the plane shrinks by cells, so it keeps its size only when the window has width 1. The plane keeps all 16 cells only when no pooling is applied. This sentence is added here and is not printed in the book.

Refining the guess with a recurrent network

One strategy for pixel labeling is to make an initial guess of the labels, then refine it using the interactions between neighboring pixels. Repeating the refinement is the same as using the same convolutions at each stage. Sharing the weights across the stages gives a recurrent network (Pinheiro and Collobert, 2014, 2015). The book shows this in figure 9.17.

Figure 9.17 unrolls the network over three steps, labeled (1), (2) and (3). The image feeds the hidden values through the kernels . The kernels turn the hidden values into an estimate of the labels. On every step after the first, the kernels are convolved over the previous estimate and added to the hidden layer. On the first step that term is zero. The same parameters are used on every step, and the estimate can be refined as many times as you wish.

• computes the hidden values from the image .
• produces the label estimate from the hidden values.
• feeds the previous estimate back into the hidden layer, and it is zero on the first step.

The demo below is a toy with one row of nine pixels. Its refinement rule is added here and is not printed in the book. It is written in the same shape as the figure: each pixel reads its own evidence and its two neighbors, and the neighbor weight plays the role of .

The labels stop changing when no pixel flips in a step. Here the neighbor weight decides whether the noisy third pixel joins its neighbors. Too large a weight also lets the boundary move, which is the same problem the next section shows with averaging.

Smoothing pixels into regions

Once each pixel has a prediction, other methods can turn the predictions into regions (Briggman et al., 2009; Turaga et al., 2010; Farabet et al., 2013). The general idea is to assume that large groups of contiguous pixels tend to be associated with the same label. Graphical models describe the relationships between neighboring pixels. The convolutional network can also be trained to approximate the graphical model objective (Ning et al., 2005; Thompson et al., 2014).

The demo uses the simplest version of this idea. Each pixel is averaged with the pixels within a radius of it, and the average gives the new label. The book does not give this averaging rule, so it is added here and is not printed in the book.

A radius that is too wide can also move a true boundary. Pixels near an edge average over more pixels from the other side, so they can change label even when they were right. The radius controls the trade-off between removing noise and keeping edges.

What to take away

• A pixel labeler emits , with one probability for each class at each pixel.
• Pooling with a large stride shrinks the output plane. The book names three ways to keep it large: avoid pooling, emit a lower-resolution grid of labels, or use unit stride.
• Refinement repeats the same convolutions, so the network is recurrent. The kernels feed the previous estimate back into the hidden layer.
• Large groups of contiguous pixels tend to share a label, so averaging over neighbors can turn noisy pixels into regions, but a wide window can also move a boundary.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 9.6 (pages 358–359), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.