Manifold Interpretation of PCA

Probabilistic PCA as a flat Gaussian

Probabilistic PCA is a linear factor model with Gaussian noise. The book describes the model it defines as a pancake: a Gaussian that is very narrow along some axes and wide along others. It is flat along its vertical axis, so its variance in that direction is small, and wide along its horizontal axes, where the variance is large. Figure 13.3 in the book shows this shape.

The wide directions carry the signal, and the narrow direction is treated as noise. PCA aligns the pancake with a linear manifold, which is a flat subspace of the data space that the signal directions span. The demo below lets you flatten the pancake and watch the eigenvalues change.

The readout prints the three eigenvalues of the sample covariance. Two of them stay large because the wide directions keep their spread. The third shrinks as the pancake flattens. The next sections show that this third eigenvalue is exactly the reconstruction error of a two-dimensional code.

Encoder, decoder and reconstruction error

The book takes a linear autoencoder as the starting point, and notes that the pancake view applies to any linear autoencoder, not only to PCA. The encoder maps an observation to a low-dimensional code, and the decoder maps the code back to a reconstruction. The encoder is

where is the mean, and the decoder is

The reconstruction error is the expected squared distance between an observation and its reconstruction:

The book states that this error is minimised when , , and the columns of form an orthonormal basis (unit length, mutually perpendicular) that spans the same subspace as the principal eigenvectors of the covariance :

The demo below fixes the mean at the centre of a two-dimensional data set. The slider sets the direction of the one-dimensional encoder and decoder. Each grey segment joins a point to its reconstruction, so its length is the error for that point.

The optimal line is the principal direction of the data, which is the eigenvector of with the largest eigenvalue. The code is the coordinate of each projected point along that line.

The error is the discarded eigenvalues

The eigenvalues of the covariance are real and non-negative. The book notes that each eigenvalue of is the variance of in the direction of its eigenvector. When has dimensions and has dimensions, with , the optimal reconstruction error is the sum of the discarded eigenvalues:

If the covariance has rank , the eigenvalues from to are zero, and the reconstruction error is zero. The demo below checks this identity on a three-dimensional pancake. It computes the error directly from the reconstructions and compares it with the sum of the discarded eigenvalues.

The two numbers agree to machine precision at every setting. That is the content of Equation 13.23. The identity holds for any data set, so the eigenvalues alone fix the optimal error for each code dimension.

Maximising variance gives the same subspace

The book notes a second route to the same solution. Instead of minimising the reconstruction error, one can maximise the variances of the elements of under an orthogonal . For one unit direction , the code variance is and the error is the variance that is left over. For centred data and an orthonormal direction these two add up to the total variance, so maximising one minimises the other.

The captured variance peaks at the same direction where the error bottoms out, because the two curves always sum to the flat total. The book presents the two views as equivalent, and the demo shows the identity that connects them: captured variance plus error equals the total variance at every angle.

The book ends the linear part of the chapter with a remark on scope. Linear factor models are some of the simplest generative models and the simplest models that learn a representation of data. They can be extended to autoencoder networks and deep probabilistic models that perform the same tasks with a more flexible model family.

Summary

• Probabilistic PCA defines a flat Gaussian, a pancake, with small variance along the thin direction. PCA aligns the pancake with a linear manifold.
• A linear autoencoder has encoder h = W^T(x - mu) (Equation 13.19) and decoder x-hat = b + V h (Equation 13.20). The minimum reconstruction error uses V = W, b = E[x], and the principal eigenvectors (Equations 13.21 and 13.22).
• The optimal error is the sum of the discarded eigenvalues (Equation 13.23), so a covariance of rank d gives zero error when the code has d dimensions.
• Maximising the variance of the code under an orthogonal encoder gives the same subspace, because captured variance and error add up to the total variance.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 13.5 (pages 499–501), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.