How to use this pageClick any colored symbol for a short explanation. Then drag every point, press every button and type into every grid you see.
the matrix singular vectors singular values eigenvalues sizes and plain vectors
Section 1
A factorization for every matrix
Push a circle through any matrix and out comes an ellipse. Read that ellipse and you read the matrix.
Drag the two rings below. They sit where the arrows and land, and those landing points are the two columns of .
The ellipse behind every matrix
and and unit circle, and the columns of (drag the rings)
Challenge +8 XPFlatten the ellipse into a line segment: make the second singular value (nearly) 0 by dragging the rings.
The semi-axes of the ellipse are the singular values: for the long one and for the short one. The directions of the axes are the left-singular vectors. The right-singular vectors are the circle points that land on the axes.
The Rotation preset gives a circle with both singular values 1, because a rotation never stretches. The Shear preset has no eigendecomposition (Section 2.7), yet its SVD has singular values 1.62 and 0.62.
Challenge +8 XPMake the ellipse a circle of radius 2: both singular values equal to 2.
Picture a camera operator handling a flat photo. Any linear move is three steps: turn the photo, zoom along two perpendicular axes by different amounts, then turn it again. That is the SVD.
The eigendecomposition of Section 2.7 has gaps. It needs a square matrix, its eigenvalues may be complex, and a shear has only one eigenvector direction. Data tables are rarely square: 1,000 examples with 20 features make a 1,000 by 20 matrix.
The singular value decomposition (SVD) fills those gaps. It finds singular vectors and singular values for every real matrix, whatever its shape.
Section 2
The three factors
First, the eigendecomposition of Section 2.7 (Equation 2.40), which the book restates as Equation 2.42 below, writes with a matrix of eigenvectors and a vector of eigenvalues.
Click any coloured symbol to see what it means.
The SVD is similar, except becomes a product of three matrices:
Click any coloured symbol to see what it means.
Say has rows and columns. Then is , is and is . Pick a shape below, then guess how many singular values it has.
Sizes of the three factors
Challenge +8 XPGuess the number of singular values correctly three times in a row. Press New shape after each answer.
The diagonal entries of are the singular values, sorted from largest to smallest and never negative (the book does not state that convention here). The columns of and are the left- and right-singular vectors. has diagonal positions, so there are that many singular values.
- and are orthogonal matrices (Section 2.6). Their columns are orthonormal, so each one keeps lengths and angles. In two dimensions that means a rotation or a mirror.
- is diagonal but not necessarily square, with the shape of .
Predict first
Take a right-singular vector and apply . Does the output point along the same line as the input?
Column by column, the three factors say this (added here, not printed in the book): sends the -th right-singular vector to the -th left-singular vector, stretched by .
Click any coloured symbol to see what it means.
Compare . An eigenvector goes to a multiple of itself. A singular vector goes to a multiple of a different vector, which may live in another space.
Section 3
Rotate, stretch, rotate
Think of an F-shaped stamp pushed through three machines: a turntable, a stretcher, another turntable. The turntables never change a size. Only the middle machine does. Read the product from the right, as it acts on an arrow. Press Play, or drag the ring to send a point of your own through the moves.
Apply the matrix as turn, stretch, turn
startcurrent frame and your point’s path and its image and its image
Challenge +8 XPMake the F come out mirrored and stretched. Slide to the end with a matrix whose determinant is negative and whose largest singular value is at least 1.5.
- turns the input space so the right-singular vectors lie on the axes (it may also mirror).
- stretches each axis by its singular value, and adds or drops axes if the matrix is not square.
- turns the result into the output space.
This mirrors from Section 2.7. The outer factors are orthogonal, so a transpose replaces the inverse.
Predict first
The Rank 1 matrix has singular values and 0. Does the F keep its shape after all three moves?
In the first third the shape turns, and maybe flips, but keeps its size. In the second third it is pulled along the axes by and , the only move that distorts it. In the last third it turns again. The colored arrows start as right-singular vectors and finish as and . The frames in between are blends for animation, and the three endpoints are exact.
Section 4
The link to eigendecomposition
Products of a matrix with its transpose have the squares of its singular values as eigenvalues. The book reads the SVD through these products. Three facts:
- The left-singular vectors of are eigenvectors of .
- The right-singular vectors of are eigenvectors of .
- The non-zero singular values of are the square roots of the eigenvalues of . The same holds for .
The last fact in one line:
Click any coloured symbol to see what it means.
Going deeper: where the facts come from
Added here, not printed in the book. Put into . A transpose reverses the order, so , and the middle is the identity because is orthogonal:
Click any coloured symbol to see what it means.
The matrix is symmetric. The right side has exactly the shape of Equation 2.41: an orthogonal , a diagonal with the squared singular values on its diagonal, and the transpose. So holds the eigenvectors of and the are its eigenvalues. Reverse the order of the factors to get the left side:
Click any coloured symbol to see what it means.
The matrix is and is . They share the non-zero eigenvalues. The bigger one has extra eigenvalues equal to 0. Test the claim: type numbers into the grid, or press New random matrix.
Singular values against eigenvalues
Challenge +8 XPOn the 2 by 2 tab, type a matrix whose singular values are exactly 4 and 3. (Think about which matrix has a very easy SVD.)
Each row compares a singular value with the eigenvalues of the two symmetric products. The squares match the eigenvalues of and of wherever both exist. The last two columns show how well singular vectors line up with eigenvectors: 1.000 means the same line. Where an eigenvalue is 0 or repeated the eigenvectors are not unique (Section 2.7), and the column shows a dash.
Section 5
Generality and the size of singular values
Make a call on each matrix below before the answer shows. The book sums it up: every real matrix has an SVD, but the same is not true of the eigendecomposition. If a matrix is not square, the eigendecomposition is not even defined, and you must use the SVD.
Which factorization exists?
The rotation has complex eigenvalues, yet singular values 1 and 1: it never stretches. The shear has only one eigenvector direction, yet singular values 1.62 and 0.62. Tall and wide matrices have no eigendecomposition by definition, and the SVD gives them singular values anyway. The rank 1 matrix has both, with one zero eigenvalue and one zero singular value, so it is singular.
Challenge +8 XPCall the eigendecomposition correctly for three different matrices.
Reading the singular values
Singular values tell you how much of a matrix lives in each direction. Think of a song mixed from loud and quiet tracks: mute the quiet ones and you still hear the song. Keep only the first terms, written one direction at a time:
Click any coloured symbol to see what it means.
This is added here and is not in the book section. The truncation is the best rank- approximation in the sum-of-squares sense, and its error is the root of the sum of the discarded squared singular values.
Keep only the largest singular values
The demo builds a 14 by 14 grid from three smooth patterns plus noise (a seeded formula, not a measured image). Click a bar to keep that many terms.
The bars drop steeply after the third value, because three patterns generated the data. Three terms already look like the original, and what is thrown away is the noise. The readout compares the true error with the formula from the discarded singular values, and they agree. All 14 terms reproduce the grid exactly. A zero singular value, as in the rank 1 matrix, is the extreme case: that direction carries nothing.
Challenge +8 XPFind the smallest that keeps at least 99% of the grid’s total squared size. Click a bar or use the slider.
The book also points ahead: the next section builds the Moore-Penrose pseudoinverse from this factorization, because and are easy to invert and can be inverted entry by entry wherever its entries are non-zero.
What to take away
- Every real matrix factors as with orthogonal , diagonal and orthogonal .
- The diagonal entries of are the singular values. The columns of and are the left- and right-singular vectors.
- Geometrically the matrix turns, stretches along the axes by the singular values, and turns again. The unit circle becomes an ellipse with those semi-axes.
- Left-singular vectors are eigenvectors of , right-singular vectors are eigenvectors of , and the non-zero singular values are the square roots of their non-zero eigenvalues.
- Unlike the eigendecomposition, the SVD exists for every real matrix, including rectangular ones, and it is the basis for generalizing inversion to them.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.8 (pages 44–45), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.