Dataset Augmentation

Making more examples that keep their label

Think of a piano student who practices every scale in every key. The student has not bought new music, but has made many more exercises from the same notes. Dataset augmentation does the same for a model: it creates new training examples from the ones already in the set, so the model sees more variety.

The book says this approach is easiest for classification. A classifier must map a complicated input to one category , so the task is to be invariant to many changes of the input. If we change in a way that leaves the category the same, the new pair is a valid example. The rule can be written as:

Here is a transformation of the input, such as a shift of a few pixels. This statement of the rule is added here and is not printed in the book. The rule is only as good as the transformation: a transformation that changes the correct class breaks the rule, however much data it adds.

The book also names invariances that are hard to build in. Turning a three-dimensional object out of the picture plane cannot be done as a simple geometric operation on the pixels. Augmentation has also been effective for speech recognition (Jaitly and Hinton, 2013).

Predict first. A horizontal flip turns a lowercase b into which letter?

Choose an answer, then flip the glyph in the game.

The glyph game has two kinds of move. A shift keeps the shape, so the letter or digit stays the same. A mirror or a half turn changes the shape into a different glyph. Watch the readout when you press a move that changes the glyph.

Checkpoint. Why is a horizontal flip not a valid augmentation for the letters b and d?

Answer, then check the game.

Noise in the input

Injecting noise into the input can also be seen as a form of augmentation. For many classification tasks, and some regression tasks, the answer should still be right when a little random noise is added. The book notes that neural networks are not very robust to noise, so training with random noise on the inputs is one way to improve robustness. Noise can also be added to hidden units, and dropout can be seen as multiplying by noise. Noise on the hidden units can be highly effective, provided its magnitude is carefully tuned.

Think of a radio station heard in a car. A little static does not change what is being said, so a listener who learns to follow speech through static copes better. The noise teaches the model to ignore the static and keep the message.

A noisy input is written as follows. This equation is added here and is not printed in the book. The noise is drawn from a Gaussian with spread , and the label stays the same:

The demo uses one input on each side of a boundary at zero. The class 1 inputs sit at and the class 0 inputs at . Each noisy input spreads around its center with width . The shaded parts that cross the boundary are the inputs that land on the wrong side.

The share that crosses is for each class, where is the cumulative Gaussian. This share formula is added here and is not printed in the book. It is derived here only for this one-dimensional picture.

Predict first. The noise spread grows from 0.3 to 1.5. What happens to the share that crosses?

Choose an answer, then move the noise spread.

Checkpoint. With a noise spread of 1 and a distance of 1, what share of inputs crosses?

Answer, then check the demo.

Sorting the operations

The book separates two kinds of input change. Operations that are generally applicable, such as adding Gaussian noise to the input, are usually treated as part of the learning algorithm. Operations that are specific to one application domain, such as randomly cropping an image, are usually treated as separate pre-processing steps.

Think of a chef and a kitchen assistant. The chef's method is the same for every dish. Washing and cutting the vegetables for one particular dish is the assistant's job. Some cases are less clear than others.

Predict first. Is adding Gaussian noise to every input part of the algorithm, or pre-processing for one domain?

Choose an answer, then sort the operations below.

The book's rule of thumb is about how widely an operation applies. Noise can be applied to any input, so it belongs to the algorithm. Cropping and rotating depend on the photographs being images, so they are pre-processing. The book names cropping directly. The placement of rotation follows the same rule and is added here and is not printed in the book. Where a case is unclear, state which side you chose and why.

Checkpoint. Under the book's rule of thumb, random cropping of photographs is usually treated as:

Answer, then check the list.

A fair comparison

The book warns that when two algorithms are compared, both must be evaluated with the same hand-designed augmentation. If one algorithm gets the augmentation and the other does not, the gain may come from the augmentation, not from the algorithm. Controlled experiments are needed to separate the two.

Think of two runners in a race where one wears running shoes and the other wears flippers. Who wins tells you about the shoes, not about the runners.

Predict first. Algorithm B is slightly better than A without augmentation. If only B is given the augmentation, how much of the gap is due to the augmentation?

Choose an answer, then tick the boxes below.

The book notes that deciding whether an experiment is properly controlled sometimes requires subjective judgment. The fair rule is the same either way: report which augmentation each algorithm received, and use the same one for every algorithm in the comparison.

Checkpoint. If only algorithm B is augmented, the gap between the two mostly measures:

Answer, then check the bars.

• Dataset augmentation makes new examples by changing inputs in ways that keep their label. A change that alters the label is not a valid augmentation.
• Adding noise to the input is an augmentation. Its effect on the crossing share grows with the spread of the noise.
• Generally applicable operations are usually part of the algorithm. Domain-specific operations are usually pre-processing.
• Compare algorithms only when they share the same augmentation, or the gain may belong to the augmentation.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.4 (pages 240–241), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.