MathalyseResearch

Learn AI · Chapter 2 · Linear Algebra

The Trace Operator

Add up a diagonal and gain a tool: write the Frobenius norm without sums, move factors around inside a trace, and rewrite a quadratic form as a trace.

Book §2.10, pages 46–47Equations 2.48–2.53About 30 minutes
Section 1

Adding up the diagonal

How to use this page

Click the colored symbols to see what they mean. Type into the grids, and commit to a guess in each Predict first box before you read the answer.

Imagine a square tally sheet. Only the cells where the row number equals the column number count. Every other cell is scenery.

A square matrix has as many rows as columns. Its main diagonal runs from top-left to bottom-right, and the trace adds up that diagonal:

(2.48)
Click any coloured symbol to see what it means.

Here is the entry in row and column , and the sum visits every row.

, , , matrices and data , second matrix, direction , trace, Frobenius norm
Predict first

A 3 by 3 matrix has diagonal 2, 3 and 6, so its trace is 11. You change one off-diagonal entry from 4 to 40. What happens to the trace?

The trace of a matrix

Type numbers into the grid. The teal cells are the only ones that count. Tick the box to fade the rest.

Challenge +8 XP

Make the trace exactly 0 by typing numbers into the grid.

Challenge +8 XP

Open the 2 by 2 tab and make the two eigenvalues add up to 10. Only the diagonal needs to change.

The 1 by 1 tab shows the book’s remark that a scalar is its own trace:

The right-hand matrix in the demo is the transpose : entry moves to . A diagonal entry has , so it stays put. The trace ignores the transpose:

(2.50)
Click any coloured symbol to see what it means.
Going deeper: the trace as a sum of eigenvalues (added here)

The trace equals the sum of the eigenvalues . The 2 by 2 tab shows it: the readout lists both eigenvalues, from the quadratic formula, and their sum matches the trace. For a complex pair the imaginary parts cancel.

So a sum over the diagonal reveals a total over the whole spectrum without computing a single eigenvalue. The book gives the practical reason next: the trace turns summation notation into matrix products.

Section 2

The Frobenius norm as a trace

Picture a chessboard where each entry of a matrix is a square whose area is the entry squared. The Frobenius norm is the square root of the total area.

Predict first

You flip the sign of every entry of a matrix. What happens to the Frobenius norm?

A double sum squares every entry and adds them all. The trace writes it with none:

(2.49)
Click any coloured symbol to see what it means.
Going deeper: why the trace gives the sum of squares (added here)

Entry of is row of times itself: the sum of the squares of that row.

Every entry of lands in exactly one row, so the trace adds each square once. The check in this block is added here and is not printed in the book.

Squares on the diagonal

Drag up or down inside a cell to change that entry. The square grows or shrinks, and the sums follow.

Set the entries with sliders

Each row’s squares add up to one diagonal entry of . The matrix groups the same squares by column, so its diagonal differs but its trace is the same total. An entry of 0 removes its square.

Its sign does not matter, since and give the same area. The trace form lets identities rearrange the sum, and the next section supplies them.

Challenge +8 XP

Make the Frobenius norm equal to 5 (between 4.95 and 5.05) by dragging or sliding the entries.

Challenge +8 XP

Make the two rows equally long: their squared lengths within 0.1 of each other.

Section 3

Moving factors around a trace

Thread three beads on a ring: , , . The trace cannot see where the ring is cut, so rotating it changes nothing.

That is the cyclic rule. For three factors the book writes

(2.51)
Click any coloured symbol to see what it means.

and for a product of matrices , moving the last one to the front gives

(2.52)
Click any coloured symbol to see what it means.

Repeat the move and any factor reaches the front.

The shapes only need to fit: the rule holds even when the rotated product has a different shape. The smallest case has two factors, of shape and of shape :

(2.53)
Click any coloured symbol to see what it means.
Predict first

is 2 by 5 and is 5 by 2. Then is 2 by 2 and is 5 by 5, two different squares. Will their traces agree?

Tr(AB) and Tr(BA) side by side

Choose the shapes. Click any number in or to raise it by 1 and watch both traces move together. From 4 the number wraps to −4, so every small value can be reached.

Challenge +8 XP

Make exactly 0 by clicking numbers in and . will follow.

The product grid shows why. Its entries are the pieces of the double sum:

Row sums are the diagonal of . Column sums are the diagonal of . Both cut the same pile of numbers, so the grand totals agree.

Predict first

Of the orders , and , which one is a rotation of ?

Allowed reorderings of factors

Here is , is and is . The rotations are always defined and always agree. Other orders such as are different rings, defined only when shapes allow, and usually different.

Rotations versus other orders

Set the shapes, then click any number in the three matrices. Each click adds 1, and 4 wraps to −4.

OrderKindShapeTrace
Challenge +8 XP

Break the rule: click numbers in , or until one of the other orders gives a different trace from the rotations.

When the other orders agree with the rotations, it is a coincidence of the entries, not a rule.

Section 4

The trace applied to a direction

This section is added here, not in the book. It uses the trace on the problem of Section 2.12: find the single direction that best summarizes a cloud of points.

The rows of are the points, and is a unit vector, so . A scalar is its own trace, and the cyclic rule moves to the end. So a quadratic form becomes a trace:

With , the squared error for a unit is:

Predict first

For a unit , the error is a constant minus the quadratic form . The form grows. What does the error do?

The error equals a constant minus a quadratic form. So minimizing the error means maximizing the form. The demo checks every step on a small seeded data set.

One identity, five ways to compute it

Drag the arrow to turn , or click the chart. Orange segments are the errors left after squashing each point onto the line.

errorquadratic formtotal

The lines marked and always match: that is the cyclic rule plus the scalar rule. The Frobenius line matches the constant minus the form, which is the whole identity. The two curves are mirror images, so where the form is large the error is small.

Challenge +8 XP

Turn to the angle with the smallest error (within 2 degrees). The error curve has one bottom.

  • The trace adds the diagonal entries of a square matrix. Off-diagonal entries and the transpose do not change it, and a scalar is its own trace.
  • : the diagonal of holds the squared row lengths, and the trace adds them.
  • The trace is invariant under cyclic permutations, , even when the shapes of the products differ. A general reordering is not allowed.
  • These rules turn quadratic forms and squared norms into traces, which prepares the derivation of PCA in Section 2.12.
Final quiz

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.10 (pages 46–47), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.