Probabilistic PCA and Factor Analysis

The big picture

A linear factor model describes data in two steps. First, a short list of hidden causes is drawn. Second, each observed measurement is a fixed linear mix of those causes plus a little noise. Think of a mixing desk. A few hidden knobs, the factors, set the level of every channel. Each channel is one observed measurement, and the hiss on each channel is its noise.

The first step draws the factors from a factorial distribution. Factorial means each factor is drawn independently of the others, so the joint distribution is a product of one-factor distributions:

The second step computes the observations. Each observation is a weighted sum of the factors plus an offset and noise:

Here is the list of latent factors (latent means hidden), one number per factor, and is the list of observed values, one per measurement. The matrix is the loading matrix. Its entry in row and column says how strongly factor pushes measurement . The vector is an offset. The noise is typically Gaussian and diagonal, which means each measurement gets its own independent noise.

The book draws this as a graph with arrows from factors to observations (its Figure 13.1). The demo below rebuilds that picture. Type the loadings, move the factors and watch the observations change.

Factor analysis

Probabilistic PCA, factor analysis and other linear factor models are special cases of Equations 13.1 and 13.2. They differ only in the noise distribution and in the prior, which is the distribution of the factors fixed before any data are seen.

In factor analysis, the factors get the simplest possible prior, the unit-variance Gaussian of Equation 13.3. The observations then get conditionally independent noise. Conditional independence given means that once the factors are known, the noise on one measurement tells you nothing about the noise on another.

The noise covariance is diagonal, , so each measurement has its own variance. The entries of are the per-variable variances, one for each observed value.

To marginalise the factors means to remove them by summing, or for continuous factors integrating, over every value that can take. The result is a Gaussian for the observations, with a covariance that combines a factor part and a noise part. The factors are the reason the observations are correlated with one another. All correlation travels through , and the noise adds only to the diagonal:

Read the covariance as two parts. The matrix carries the shared structure that the factors create. The diagonal carries each measurement’s private noise. The covariance of the observations is the sum of the two.

The demo draws the covariance as an ellipse. Its size and tilt come from . The diagonal noise adds to the two diagonal entries, so it changes the width along each measurement axis but cannot create correlation on its own. A tilted factor creates a tilted ellipse, which means correlation between the two measurements.

Probabilistic PCA

Probabilistic PCA makes one change to factor analysis. It gives every measurement the same noise variance, so becomes a multiple of the identity, . The noise is then the same in every direction, a property called isotropic noise:

Equivalently, the model is written as a single line with one new noise term, where is a standard Gaussian noise vector:

The book cites Tipping and Bishop (1999), who give an iterative EM algorithm for estimating and from data. This page does not train the model. It shows what the model says about data once and are known.

The motivation is that most variation in the data is captured by the latent variables , up to a small residual reconstruction error of size . The residual is the part of that the factors do not explain.

As Tipping and Bishop (1999) show, probabilistic PCA becomes PCA as goes to zero. This is the most important result of this section. The cloud collapses onto the span of the columns of , meaning every weighted sum of those columns (a line when there is one column), and the thin direction disappears. The demo lets you watch this happen.

In the demo the cloud is stretched along the loading direction by and across it by . The loading vector has length , fixed at 1.5 here. Watch the thin direction as falls.

The sharp limit and the projection

The book makes two statements about the limit. As goes to zero, the density becomes very sharp around the span of the columns of . Also, the conditional expected value of given becomes an orthogonal projection of minus onto that span. An orthogonal projection is the point on the span closest to the data point. It is found by dropping a perpendicular from the point onto the span.

The book states the projection without a formula. The formula below is added here and is not printed in the book. It follows from Gaussian conditioning on Equations 13.5 and 13.6:

Each entry of is a coordinate, one per factor, of the point on the span that best explains . As goes to zero, the term that damps the matrix vanishes, and the coordinates become the ordinary least-squares coordinates. Least squares picks the orthogonal projection, so the limit is the projection the book names.

The second statement is a warning. A sharp density gives very low likelihood to points that are not near the span. The log-likelihood is the logarithm of the probability density, so a larger value means a more likely point. The log-likelihood of a point under Equation 13.5 is

Here is the covariance from Equation 13.5. This formula is added here for the demo and is not printed in the book.

Summary

• The factors are drawn from a factorial prior. The observations are a linear mix of them plus noise, Equations 13.1 and 13.2.
• Factor analysis gives the observations the covariance , where is diagonal and holds each measurement’s private noise.
• Probabilistic PCA sets all noise variances equal to , which makes the noise isotropic.
• As goes to zero, the model becomes PCA, and the conditional mean of becomes an orthogonal projection. A point far from the span then has very low likelihood.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 13.1 (pages 490–491), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.