Section 1
An operation and its undo
How to use this pageClick any colored symbol to read it. Then drag, type and guess before each answer.
Every useful operation has an undo: stretch a photo, then shrink it back. Can a matrix be undone too?
the operation
its undo
, vectors and the do-nothing matrix
the known right side
A system (Eq. 2.11) asks for the input behind a known result : an undo. The book calls the tool matrix inversion, and says it solves the system “for many values of ”, not all of them.
Below, a 2 by 2 matrix moves the plane, and the right picture applies the candidate undo, . Drag the two colored dots: they are the columns of .
Apply a matrix, then try to undo it
Set the four entries with sliders
original FF after AF after the undo
Challenge +8 XPDrag the dots until the middle picture is squashed flat onto a line. The readout should say there is no undo.
For most settings the right-hand F lands exactly on the left-hand one: the second matrix really is an undo. Identity is the do-nothing matrix. Rotate and Stretch are easy to reverse.
A squashed matrix is the instructive case. It pushes the whole plane onto a line, so many different points land on the same spot, and nothing can pull them apart again. The next section of the book (2.4) gives the exact condition. Here the focus is what an inverse is, and how to use it.
Section 2
The identity matrix
An inverse needs a matrix that does nothing: the identity matrix. For vectors with entries it is , square with shape , and formally
Click any coloured symbol to see what it means.
The symbol reads “for all”. Whichever you pick, returns it.
The structure is plain: ones on the main diagonal, zeros everywhere else. The book’s Figure 2.2 shows :
Click any coloured symbol to see what it means.
Why does that do nothing? Row of the identity has a single 1 at position , so it copies entry of the vector. Each row copies one entry.
Going deeper: the dot product in each row
Entry of a product is the dot product of row with the vector. For the identity this is .
Now you build one. Click a cell to cycle its value through 0, 1, 2. Rows that spoil the vector turn orange.
Build the identity
M, the matrix you editxM x (orange rows differ from x)
Challenge +8 XPClick cells until is the identity of size 3 or more: ones on the diagonal, zeros elsewhere.
With the right matrix, is a copy of for every size and every row, which is Equation 2.20. A scrambled matrix can still match one particular by luck, so draw a new vector to catch it. The identity is the one matrix that works for every .
Section 3
The inverse matrix
The matrix inverse of is written . The book defines it as the matrix with
Click any coloured symbol to see what it means.
Apply , then , and the combined effect is the do-nothing matrix. That is the right-hand picture of the first demo.
Two details. stands on the left: matrix multiplication is not commutative, so the side is part of the definition. And the inverse is for square matrices, where the book states in Section 2.4 that the left and right versions agree, so also (Eq. 2.29).
Find an inverse yourself
For a 2 by 2 matrix there is a formula. It is added here and is not printed in the book. Write the rows as and . Swap and , flip the signs of and , and divide by the number :
Click any coloured symbol to see what it means.
Try it. The matrix below is fixed. Type four numbers for until the product equals the identity.
Your inverse against the definition
Challenge +8 XPType the entries of so that is exactly the identity.
Going deeper: checking the formula
Multiply the original by the proposed inverse before the division:
The result is the identity scaled by , so dividing by it gives exactly . The same formula shows what breaks when : the division is impossible and no inverse exists. That number is the determinant, a topic of the book’s Section 2.11.
The calculator below handles sizes 2 and 3 for any entries (it uses Gauss-Jordan elimination, a general method not covered by the book) and checks the definition numerically. The slider s walks a matrix towards singular: it loads the rows (1, 2) and (2, 4 + t) with t = 10−s, so at s = 6 the second row is almost exactly twice the first.
Compute an inverse and test the definition
Challenge +8 XPUse the slider to make an inverse that contains an entry larger than 1000, while the matrix still has an inverse.
For most matrices both products come out as the identity, up to rounding errors near 10−16. Singular hits a zero pivot and reports that no inverse exists. Slide towards a singular matrix and the entries of the inverse blow up. That growth is the seed of Section 5.
Section 4
Solving Ax = b with the inverse
The payoff is a short argument. Start with the system . Multiply both sides on the left by . The left side becomes , and the definition turns into , which does nothing to :
Click any coloured symbol to see what it means.
Line 2.23 multiplies both sides by the same matrix from the same side, which keeps the equation true. Line 2.24 uses Equation 2.21. Line 2.25 uses Equation 2.20 and is the solution: . The side matters: goes on the left of , and so on the left of .
Run the argument with real numbers. Press Next step to reveal one line at a time. The table applies the same inverse to other right-hand sides.
The argument, with numbers
| Right side b | Solution | Check Ax |
|---|
Challenge +8 XPType a right-hand side so that the solution comes out as x = (1, 1).
The book makes two remarks. The argument needs to exist, which the next book section settles. And in theory one inverse solves the equation for many right-hand sides. The table shows it: each new right side costs one multiplication, and (1, 0) and (0, 1) return the columns of .
Section 5
The inverse as a theoretical tool
Now a warning for software. The book calls “primarily useful as a theoretical tool”, to be avoided in most software. The reason: a digital computer represents with limited precision, while algorithms that use directly can usually estimate more accurately.
A computer stores a real number with a fixed number of binary digits. A 32-bit (single precision) number keeps about 7 decimal digits, a 64-bit (double precision) number about 16. Some matrices magnify those rounding errors, for example matrices whose inverse has huge entries.
Going deeper: the condition number
The condition number measures the magnification: a relative error of 10−7 in the input can become an error of 10−7 times the condition number in the answer. The book treats it in Section 4.2. It appears here to explain the demo.
The demo tests this advice on a toy: the Pascal matrix, whose binomial entries are exact integers. It gets harder as the size grows. The true answer is .
Two routes run in the same arithmetic. The inverse route computes by Gauss-Jordan elimination and multiplies it with . The direct route eliminates on the system itself and never forms . Both use the same pivoting rule. Every operation is rounded to 32 bits, or left at 64 bits.
Inverse route against direct route
inverse routedirect route (no inverse)solid = error in x, dashed = residual, hollow circle on the floor = exactly 0
Challenge +8 XPWith 32-bit rounding on, find the smallest matrix size at which the inverse route’s residual is at least 100 times larger than the direct route’s.
The error is the relative distance of the computed from the true one. The residual is the length of relative to the length of : how far the computed misses the equations.
Going deeper: reading the errors and residuals
With 64-bit numbers both routes keep at least 8 correct digits up to size 9. With 32 bits the error in grows fast for both, because the condition number limits it, and neither route wins every size: at size 8 the inverse route is better, at size 9 the direct route.
The residual tells the story. With 32-bit numbers, from size 6 on, the direct route satisfies the equations to about the rounding level, and the inverse route leaves a residual at least a thousand times larger. At size 4 the gap is only about 17. The stored carries rounding errors that the right-hand side cannot correct afterwards. A toy on one family of matrices is evidence for the book’s advice, not a proof of it.
What to take away
- has ones on the diagonal and zeros elsewhere, and for every vector (Eq. 2.20, Fig. 2.2).
- defines the inverse (Eq. 2.21), written on the left.
- If exists, solves (Eqs. 2.22 to 2.25), for any right-hand side.
- Not every matrix has an inverse. A matrix that squashes space has none.
- In software, solve with an algorithm that uses directly: the inverse loses accuracy in limited precision.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.3 (pages 36–37), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.