Special Kinds of Matrices and Vectors

Why special shapes matter

A general × matrix is numbers, and every operation touches all of them. A special matrix is a fixed spanner that does fewer jobs, faster.

• Diagonal matrices scale each coordinate on its own.
• Symmetric matrices equal their own transpose.
• Unit and orthogonal vectors are length 1 and at a right angle.
• Orthogonal matrices rotate or mirror space without stretching it.

The demo counts every multiplication or division on a real × matrix. Move .

Before you leave this demo, guess. If you double , how much more does the general inverse cost?

A general product costs multiplications, a diagonal one . A general inverse by Gauss-Jordan costs , a diagonal one divisions, an orthogonal one nothing (Section 5).

Diagonal matrices

Think of a mixing desk with one volume knob per channel. No knob touches another channel. A matrix is diagonal if and only if it is zero everywhere except the main diagonal, from top left to bottom right:

The identity matrix is one. To build a square diagonal matrix from a vector , the book writes : the entries of go on the diagonal.

A diagonal matrix only scales. Output entry is , so each coordinate of gets its own factor. The symbol is the element-wise product: multiply two equally long vectors entry by entry.

Only multiplications, not . Drag the round handle , move the knobs and , and aim the violet output at the orange target.

A knob of 1.5 stretches the shape sideways. A negative knob mirrors it. A knob between 0 and 1 compresses. A knob at 0 flattens the shape onto a line, and a flattened axis cannot be restored. That is the rule for the inverse, the book’s second claim:

The inverse exists only if every diagonal entry is non-zero, otherwise it divides by zero. Tick the checkbox: the dark outline returns to the start. Then push a knob to 0 and the inverse is gone.

Rectangular diagonal matrices

A diagonal matrix need not be square. A rectangular diagonal matrix still has for . It has no inverse, but multiplying is still cheap. scales each coordinate and then pads or trims the length. A tall appends zeros. A wide drops the last entries of .

In the wide tab, move . Nothing in the output changes: that lost information is why these matrices have no inverse.

Symmetric matrices

The transpose flips a matrix over its main diagonal: entry moves to . A matrix is symmetric if flipping changes nothing:

Entry by entry, : the upper right triangle is a mirror image of the lower left. Only of the numbers are free. This count is added here and is not printed in the book.

Symmetric matrices arise from functions of two arguments that ignore their order. The book’s example is a table of distances: is the distance from point to point , and the way back is equally long. Drag the four points, then switch on a one-way cost (think uphill walking) and watch the mirror break.

The orange cells mark every pair that disagrees. Symmetry belongs to the function that produces the entries. It can be lost as easily as it is gained.

Unit and orthogonal vectors

A unit vector has length one, that is, unit norm. The L² norm is the ordinary length of . Divide any non-zero vector by its own length and you get a unit vector with the same direction, a pure compass heading.

Two vectors and are orthogonal if their dot product vanishes:

By Equation 2.34, . For non-zero vectors the product is zero exactly when : a 90 degree angle. If both also have length 1, they are orthonormal.

Drag and until the readout says orthonormal. The dashed circle has radius 1.

In , at most non-zero vectors can be mutually orthogonal. Guess what that means in the plane.

Orthogonal matrices

An orthogonal matrix is square, with mutually orthonormal rows and mutually orthonormal columns. Think of the two axes of a picture frame: turn the frame and the axes stay perpendicular and of unit length.

Entry of is the dot product of column with column of . For the identity, every column needs dot product 0 with the others and 1 with itself. The second equality says the same for rows.

The inverse undoes a matrix, so says directly that the transpose undoes :

Hence the inverse is almost free: a relabeling of entries, no arithmetic, the third bar of the cost demo. The book’s right inverse (Eq. 2.29) equals the left one for square matrices. Module 2-4 teaches this, so it is not repeated here.

Rotations and reflections

In the plane every orthogonal matrix is a rotation or a reflection (a rotation plus a mirror flip). This classification is added here and is not printed in the book. Turn the F and watch the two arrows, the matrix columns.

The arrows stay at length 1 and 90 degrees, and the transpose brings the shape back. A determinant is one number that a square matrix produces: the matrix with rows (2, 0) and (0, 5) has determinant 10. This definition is added here and is not printed in this section of the book. Every rotation here has determinant +1, and every reflection has −1.

The warning in the definition

The book stresses this: the rows and the columns must be orthonormal, not merely orthogonal. There is no special name for orthogonal vectors of unequal length. Here a rotation has its second column stretched by . The columns stay at right angles, yet as soon as the transpose no longer undoes the matrix.

For the product is diagonal with entries 1 and . The transpose over- or under-corrects. Only would work: with its second row divided by . Remember: orthonormal columns, then the transpose is the inverse.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.6 (pages 40–42), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.