Slow Feature Analysis

The slowness principle

Some features change quickly and others change slowly. If a zebra crosses an image, the brightness of one pixel flips between black and white as the stripes pass over it. A feature that says whether a zebra is present stays on for the whole crossing. The slowness principle says that useful features change slowly over time, so a model can be trained to prefer them.

The book adds a slowness term to the training cost. Here is the time index, is the feature extractor being trained, measures the distance between consecutive feature values, and sets how strongly slowness is rewarded:

The principle predates slow feature analysis. It applies to any differentiable model trained by gradient descent, as long as the cost includes a term like Equation 13.7.

The most common choice for is the mean squared difference. Under that choice, the slowness of a feature is the mean squared change between consecutive time steps. The demo below measures that quantity for the observed signals.

The two sources are two sine waves. The slow one has period 200 steps and the fast one has period 9 steps, so the fast one changes sharply from one step to the next. Each observation is a mix of both. When the fast strength is 0, the first observation is the slow signal itself. Raising the strength makes the observation change faster, even though its underlying slow component has not changed.

The SFA objective and its constraints

Slow feature analysis (SFA) is the linear case of the slowness principle. It takes the feature extractor to be linear, , and solves an optimisation problem that asks for the slowest such feature. The book notes that SFA is not a generative model in the usual sense. It defines a linear map from input to feature space but no prior over features, so it does not define a distribution over the inputs.

The objective is the slowness of one feature, minimised over its weights:

The objective is minimised subject to two constraints. The first makes the feature have zero mean over time:

The second makes the feature have unit variance over time:

Both constraints are needed. Without the zero-mean constraint, adding a constant to every feature value gives another solution with the same slowness. Without the unit-variance constraint, all features could shrink to zero, which has zero slowness and is useless.

This step is added here for the demo and is not printed in the book. The inputs are whitened with the step from Section 13.2, so that is the identity. Then the unit-variance constraint reduces to a constraint on the length of , and the objective becomes a ratio of quadratic forms, a number of the form divided by , where is a matrix that depends on the data. The slowness of the feature is

Minimising this ratio over all directions is a generalised eigenvalue problem. Linear algebra solves it directly, without iterative training. The smallest eigenvalue is the slowest feature, and at the minimum the ratio equals that eigenvalue:

The demo below works in the whitened plane. Each angle gives a unit direction . The slowest direction is the one with the smallest value of .

When the direction is far from the slowest one, the trace swings quickly and is large. As the direction approaches the slowest one, the trace becomes smooth and falls to a minimum. The minimum is the first SFA feature.

Ordering and decorrelation

SFA features are ordered, like the components of PCA. The first feature is the slowest. To learn several features, the book adds a decorrelation constraint between every pair of features:

Without this constraint every feature would capture the one slowest signal. The constraint asks each new feature to be linearly uncorrelated with the features before it, so a second feature cannot repeat the first. The book notes that this mechanism is simple here because SFA features are linear.

In a plane with whitened inputs, two unit directions are uncorrelated exactly when they are perpendicular. The second feature is therefore forced to point at right angles to the first, and the only free choice is the first direction. The demo below shows what happens with the constraint on and off.

The constraint is a linear condition in this model. In two dimensions it fixes feature 2 completely, but in higher dimensions it only restricts feature 2 to the space orthogonal to the earlier features, and feature 2 is then chosen as the slowest feature inside that space.

Slowness through a basis expansion

SFA is usually applied to a nonlinear basis expansion of the input. The book uses the quadratic expansion, which contains every product . Because every linear feature is also a quadratic feature, the slowest quadratic feature can never be slower than the slowest linear one. The question is how much lower it can go.

The demo below shows a case where the answer is much lower. Two coordinates and each oscillate quickly, but their amplitude is modulated by a slow envelope. Their sum of squares is the square of that envelope, so it is slow even though each coordinate is fast:

With the linear basis, the best feature still carries the fast oscillation and has large slowness. With the quadratic basis, the best feature is the squared envelope, which changes slowly. The book reports that SFA with a quadratic expansion learns features with properties similar to complex cells in the V1 cortex when it is trained on small patches of natural video (Berkes and Wiskott, 2005).

The book also notes that SFA is well suited to theory. For a given environment, the functions that SFA learns can be predicted from the way the underlying factors change over time. In deep SFA, the book describes stacking linear SFA modules with basis expansions in between. In practice, deep SFA on simulated data seems to recover the predicted functions. The book also cites deep SFA trained on videos of random motion in 3-D rendered environments, which learns features like those of neurons used for navigation in rat brains (Franzius et al., 2007).

The book ends this part with a caution. The slowness principle has not yet become the basis of any state of the art application. The book speculates that the slowness prior may be too strong. A position is useful whether an object moves fast or slowly, but the slowness principle pushes a model to ignore the position of fast moving objects. The book suggests a better prior would be that features are easy to predict from one time step to the next. Deep SFA has also been used for object recognition and pose estimation (Franzius et al., 2008).

Summary

• The slowness principle adds a penalty for fast change, Equation 13.7, to the training cost of any differentiable feature extractor.
• Linear SFA minimises the mean squared change of one feature, subject to zero mean and unit variance (Equations 13.8 to 13.10). In whitened space the problem is a ratio of quadratic forms.
• Features after the first must be decorrelated from the earlier ones (Equation 13.11). Without this, every feature learns the same slowest signal.
• SFA is linear and so can be solved in closed form. It defines no prior over features, so it is not a generative model.
• A basis expansion, such as the quadratic one, lets SFA find slowness that no linear feature can express, as in a fast carrier with a slow envelope.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 13.3 (pages 493–496), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.