Independent Component Analysis (ICA)

Unmixing the room

Independent component analysis (ICA) starts from a room where several things are happening at once. Several people talk, and several microphones each pick up a different blend of all of them. Every microphone hears every voice, at a different volume. ICA tries to undo the blend and return one voice per output. The same problem arises in neuroscience. Electrodes on a scalp record a blend of heart, eye and brain signals, and ICA separates the brain signals from the rest.

In the book’s model, the sources are drawn from a prior that you choose in advance. The observations are then a fixed linear mix. The model is deterministic, with no noise term:

Learning means finding the inverse, so that the sources can be read back from the observations:

Each row of the observations is one moment in time. Each value is one sensor, and each source is one underlying signal. The demo below uses two voices and two microphones, so is a 2 by 2 table of numbers. The mixing is controlled by the two numbers and , with the diagonal fixed at 1.

The left plot is the square of the sources, which the observations hide. The right plot is that square after the mixing. Each source direction stretches into a column of , so the edges of the parallelogram point along the columns (1, ) and (, 1). ICA is the task of finding those edges from the data alone, without knowing .

Whitening and the uncorrelated trap

A common first step is whitening. This step is added here for the demo and is not printed in the book. It removes all correlation between the observations and gives them unit variance. With the sample covariance and the mean , the whitened data is

After whitening, the covariance of is the identity matrix. Whitening is a necessary step in many ICA algorithms, but it is not enough by itself. Any rotation of also has the identity as its covariance. The rotated outputs are

The book makes the point in a footnote. Uncorrelated means the covariance between two variables is zero. Independent means knowing one tells you nothing about the other, so their joint distribution factorises. Independence is much stronger. A cloud can be uncorrelated and still have a shape that reveals a dependence. The demo shows exactly that.

Whitening alone does not select a rotation, since every rotation leaves the covariance unchanged. ICA needs a criterion that changes with the angle. The rest of this module uses the excess kurtosis for that purpose.

Why the sources must be non-Gaussian

The book explains why the prior over the sources must be non-Gaussian. Once the mixing is deterministic, the density of the observations follows the change-of-variables rule of Equation 3.47 in Chapter 3, with the inverse of the mixing:

If every source is Gaussian, then rotating the sources gives Gaussian sources with the same joint distribution. Two different unmixing matrices then give the same distribution of . The book says that is then not identifiable, which means no data can single out one answer. The excess kurtosis measures how far a distribution is from Gaussian:

This definition is added here for the demo. A Gaussian has excess kurtosis 0, and a uniform distribution has −1.2. The sum of the absolute kurtoses of the outputs is high when each output is close to one source. The demo rotates the whitened data and compares uniform sources with Gaussian ones.

The prior that ICA uses is a choice. The book describes the sigmoid-derivative prior. A logistic sigmoid rises from 0 to 1, and its derivative is a density that integrates to 1:

The derivative of the sigmoid peaks at zero and has heavier tails than a Gaussian with the same variance. A prior that peaks sharply at zero favours sparse sources, meaning sources whose values are near zero most of the time. The predict step below compares the two peaks.

Variants and extensions

The book lists several variants and extensions. Most do not use maximum likelihood, which chooses the parameters that make the observed data most probable. Instead they aim to make the elements of = independent by some other criterion, such as maximising the kurtosis.

• Noise in the generation. Some variants add noise to instead of using a deterministic decoder.
• Orthogonal unmixing. Equation 3.47 needs the determinant of , which is expensive and can be numerically unstable. Some variants avoid the determinant by requiring to be orthogonal, so that its inverse is its transpose, the matrix with rows and columns swapped. The demo below shows what that constraint does.
• Nonlinear ICA. A nonlinear function can generate from , which generalises the linear model towards the autoencoders of Chapter 14.
• NICE. The nonlinear independent components estimation method stacks invertible transformations whose Jacobian determinants are cheap to compute. The Jacobian is the table of partial derivatives of a transformation, and its determinant measures how the transformation stretches volume. It gives an exact likelihood and easy sampling.
• Independent subspace analysis. Units form groups. Dependence is allowed inside a group but discouraged between groups.
• Topographic ICA. Units are placed on a spatial grid, and neighbours are encouraged to learn similar features.

The book also notes a distinction. A generative model either represents or can draw samples from it. Many ICA variants only transform between and and never represent the prior, so they are used mainly for separating signals, not for generating data.

The demo below shows the cost of the orthogonal constraint. A rotation matrix has determinant 1 and an inverse equal to its transpose. A general matrix has neither property.

Summary

• ICA models the observations as a deterministic mix = of independent sources, and learns the unmixing .
• Whitening removes correlation but does not identify the sources, because every rotation keeps the covariance at the identity. Independence needs a criterion that changes with the rotation.
• The prior must be non-Gaussian. For Gaussian sources, rotations are indistinguishable and is not identifiable.
• The sigmoid-derivative prior has a higher peak at zero than a Gaussian of the same variance, and this pushes the learned features towards sparsity.
• Constraining the mixing to be orthogonal makes the inverse equal to the transpose and the determinant equal to plus or minus 1.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 13.2 (pages 491–493), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.