Tangent Distance, Tangent Prop and the Manifold Tangent Classifier

Tangent distance

Many algorithms assume that data lies near a low-dimensional manifold. One of the early attempts to use this is the tangent distance algorithm (Simard et al., 1993, 1998). It is a nearest-neighbor method in which the distance is not Euclidean but comes from the manifolds on which probability concentrates. Examples on the same manifold share a category, so the classifier should be invariant to movement along the manifold.

The exact manifold distance requires solving an optimization problem to find the nearest pair of points on two manifolds. The book proposes a cheaper local alternative: approximate each manifold by its tangent plane at the point, and measure the distance between the two tangents, or between a tangent plane and a point. That needs a low-dimensional linear system, but the tangent vectors must be specified in advance.

The demo uses a unit circle as the manifold and measures the distance from a point to a tangent line at . Because the circle is simple, the exact distance from to the circle is known, and the check below verifies it by sampling.

Predict first. Which tangent point gives the exact distance from to the circle?

The two equations above are added here and are not printed in the book. The tangent line at on the unit circle has normal equal to itself, so the tangent distance reduces to the expression shown. The exact distance is the gap between and the circle of radius 1.

Tangent prop and its penalty

The tangent prop algorithm (Simard et al., 1992) trains a neural network classifier with an extra penalty. It makes each output locally invariant to known factors of variation, which are the directions of movement along the manifold near which examples of the same class concentrate. Figure 7.9 illustrates the idea.

Local invariance is achieved by requiring to be orthogonal to the known tangent vectors at . That is equivalent to requiring the directional derivative of in each direction to be small. The penalty is:

The regularizer can be scaled by a hyperparameter. For most networks the sum runs over many outputs rather than the single output used here for simplicity. The tangent vectors are derived in advance, usually from the known effect of transformations such as translation, rotation and scaling of images. The book notes that tangent prop has also been used in reinforcement learning (Thrun, 1995).

Predict first. If the gradient is orthogonal to a tangent vector, what is the penalty for that direction?

Figure 7.9 is a rebuild of the book’s illustration with two one-dimensional manifolds in a two-dimensional space. The rebuilt curves are drawn for this module and are not the book’s figure. The classifier should change rapidly in the normal direction and not change along the manifold.

Tangent prop against dataset augmentation

Tangent propagation is closely related to dataset augmentation. Both encode prior knowledge by specifying transformations that should not change the output. Dataset augmentation trains the network explicitly on transformed inputs. Tangent propagation does not visit any new input point. It regularizes the model analytically to resist perturbation in the specified directions.

The book names two drawbacks of the analytical approach. First, it only resists infinitesimal perturbations, while explicit augmentation confers resistance to larger ones. Second, it poses difficulties for rectified linear units. A ReLU network can shrink its derivatives only by turning units off or by shrinking its weights. A sigmoid or tanh unit can also shrink its derivative by saturating at a high value with large weights.

Predict first. A sigmoid unit with a large weight saturates. What happens to its derivative as the weight grows?

The derivatives in the demo are computed here from the standard formulas, for input 1: the sigmoid derivative is and the ReLU derivative is while the unit is active. They are not printed in the book, which states the argument in words.

Dataset augmentation works well with rectified linear units because different subsets of units can activate for different transformed versions of the same input. So for rectified linear units the book points to explicit dataset augmentation rather than the analytical penalty.

Double backprop and adversarial training

Tangent propagation is also related to double backprop (Drucker and LeCun, 1992) and to adversarial training (Szegedy et al., 2014b; Goodfellow et al., 2014b). Double backprop regularizes the Jacobian to be small. Adversarial training finds inputs near the original inputs and trains the model to produce the same output on them.

The book describes the pairs as non-infinitesimal versions of one another. Dataset augmentation is the non-infinitesimal version of tangent propagation, and adversarial training is the non-infinitesimal version of double backprop. Tangent propagation and augmentation require invariance to specified directions. Double backprop and adversarial training require invariance to all directions of change, so long as the change is small.

This is a useful way to sort the methods. The question for each method is which directions of change the model must ignore, and whether that set is specified by hand or taken as every small direction.

The manifold tangent classifier

The manifold tangent classifier (Rifai et al., 2011c) removes the need to know the tangent vectors in advance. It uses an autoencoder to learn the manifold structure from unlabeled data, and estimates the tangent directions from that. Chapter 14 describes autoencoders. The estimated tangents go beyond the classical invariances of images, because they can include object-specific factors such as moving body parts.

The algorithm has two steps: first, learn the manifold by unsupervised learning with an autoencoder; second, use the estimated tangents to regularize a neural net classifier as in tangent prop, with the penalty of Equation 7.67.

Predict first. In the manifold tangent classifier, where do the tangent vectors come from?

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.14 (pages 270–273), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.