Adversarial Training

Inputs the network gets wrong

A network can score well on a test set drawn from the same distribution as its training data and still fail on inputs chosen to fool it. The book reports that even neural networks with human-level accuracy have a nearly 100% error rate on adversarial examples. These are inputs built by an optimization procedure that searches for a point near a data point at which the model output is very different (Szegedy et al., 2014b).

The changed input can be so similar to the original that a person cannot tell the two apart, while the network predicts differently. The book notes that adversarial examples matter for computer security, but that this is beyond the scope of the chapter. Here they matter because they reveal how the network understands its task.

Predict first. The classifier below is linear, with output . The teal dot marks the point, and its label is B. Each coordinate may move by at most = 0.10. Does the label flip?

The demo uses the sign step: for label B, move each coordinate in the direction of , which raises the output as much as the budget allows. Section 2 explains why this change can be so large.

The sign step and Figure 7.8

Figure 7.8 shows the attack on GoogLeNet, a deep network trained on ImageNet. The printed caption describes the method: add an imperceptibly small vector whose elements equal the signs of the gradient of the cost with respect to the input. The book’s figure has three panels, reproduced here with the printed numbers. The equation below restates the caption in symbols and is not printed as a numbered equation.

• The original panda is classified as a panda with 57.7% confidence.
• The sign vector, multiplied by 0.007, is classified as a nematode with 8.2% confidence.
• The sum of the panda and the perturbation is classified as a gibbon with 99.3% confidence.

The demo below is a stand-in. It is not GoogLeNet and does not reproduce those numbers. It is a three-class softmax classifier on sixteen inputs, built only to show how the sign step changes the top class as the budget grows.

The sign step has one advantage over a random change. It moves every coordinate in the direction that the gradient says is most harmful, so, to first order, the change in the cost is as large as the budget allows. This first-order argument is added here and is not printed in the book. The book reports the same kind of attack on a deep network, and the stand-in shows the mechanism in a space that fits on the page.

Why small changes add up

The book explains that one of the primary causes of adversarial examples is excessive linearity (Goodfellow et al., 2014b). Neural networks are built from nearly linear blocks, and the overall function they implement is often highly linear in practice. Linear functions are easy to optimize, but the value of a linear function can change very rapidly when it has many inputs.

The book states the consequence: if each input changes by , a linear function with weights can change by as much as , which can be very large when is high-dimensional.

Predict first. A linear model has = 100 inputs, each weight has size 0.1, and each input moves by = 0.01. How far can the output move?

The derivation below is added here and is not printed in the book. It takes the perturbation that aligns each step with the sign of the weight, , and shows where the change comes from.

The aligned bound grows linearly with the number of inputs. A random direction grows more slowly, with the square root of the number of inputs. This comparison is added here and is not printed in the book.

Adversarial training and local constancy

Adversarial training trains the network on adversarially perturbed examples drawn from the training set (Szegedy et al., 2014b; Goodfellow et al., 2014b). The book reports that this can reduce the error rate on the original i.i.d. test set.

Adversarial training discourages the sensitive, locally linear behavior. It encourages the network to be locally constant in the neighborhood of the training data, which the book describes as an explicit local constancy prior for supervised neural networks.

Linear models such as logistic regression cannot resist adversarial examples, because they are forced to be linear. Neural networks can represent functions that range from nearly linear to nearly locally constant. They can therefore capture linear trends in the training data while still learning to resist small perturbations.

The demo below is a one-dimensional stand-in, not a result from the book. A linear output with slope changes across the budget band, while a saturated output with the same value at the data point stays nearly flat across it.

Virtual adversarial examples

Adversarial examples also give a route to semi-supervised learning. At a point with no label in the dataset, the model itself assigns a label . That label may not be the true label, but if the model is of high quality, has a high probability of being the true label.

We can then look for an adversarial example that makes the classifier output a label different from . Adversarial examples built from the model’s own label instead of the true label are called virtual adversarial examples (Miyato et al., 2015). The classifier is then trained to assign the same label to and .

The motivation is an assumption that different classes usually lie on disconnected manifolds, so a small perturbation should not be able to jump from one class manifold to another. The approach encourages a classifier that is robust to small changes anywhere along the manifold where the unlabeled data lies.

Predict first. Why does the method use the model’s own label rather than a true label?

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.13 (pages 268–270), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.