MathalyseResearch

Learn AI · Chapter 2 · Linear Algebra

Norms

Measure the size of a vector and a matrix with the Lp norms, know which properties make a function a norm, and choose between the squared L2, L1 and max norms.

Book §2.5, pages 39–40Equations 2.30–2.34About 30 minutes
Section 1

Measuring size and the norm properties

How big is a vector? A hiker walks 3 km east and 4 km north. As the crow flies she is 5 km from the start. On the roads she walked 7 km. A chess king needs 4 moves. Three rulers, three answers, and each ruler is a norm: a rule that turns a vector into one non-negative size.

How to use this page

Click any colored symbol, in an equation or in a sentence, and it explains itself. Then drag the vectors and move the sliders before you read the text under each demo.

a vector a second vector ‖·‖ a norm , numbers you choose

Drag the hiker’s destination and read all three rulers.

Three rulers for one trip

crow: straight linetaxi: east, then northking: largest single step
Challenge +8 XP

Drag the destination to a spot where all three rulers give the same number.

Not every rule deserves the name. A function is a norm if it maps a vector to a non-negative number and obeys three properties:

Only the zero vector has size zero. The triangle inequality says the direct path to is never longer than a detour through . Scaling a vector by multiplies its size by , whatever the sign.

Test them. The black arrow is , the violet one , the orange one . Drag the tips and read both sides of each rule.

Testing the three properties

unit ball of the norm
Challenge +8 XP

With , drag to point the same way as (within a few degrees), so that the two sides of the triangle inequality become equal.

Challenge +8 XP

Set , then drag and until the triangle inequality fails. The readout says when it does.

Going deeper: an exponent below 1

The book requires . For the formula of the next section still gives a number, but the triangle inequality breaks: and have sizes 1 and 1, yet has size 4. This counterexample is added here and is not in the book. The unit ball is pinched inward, and the straight shortcut leaves it.

Section 2

The Lp family and its unit balls

One formula builds a whole family of rulers. For a real number , the Lp norm of is

(2.30)
Click any coloured symbol to see what it means.

Take each entry without its sign (the bars), raise it to the power , add the results, and take the -th root. With you get the taxi distance, with the crow distance, and a large listens mostly to the largest entry.

The unit ball of a norm is the set of all points whose norm is 1. This definition is added here and is not printed in the book. Guess the shape first.

Predict the unit ball

Now play with yourself and drag the vector. The dashed outline is the unit ball enlarged until it passes through your vector’s tip.

Unit balls of the Lp norms

unit ball (norm = 1)all points with the same norm as the vectorthe vector (drag the tip)
Challenge +8 XP

Set and drag the tip onto the diamond, so that its norm reads 1.00 (within 0.04).

The diamond becomes a circle at and a square as grows. The table shows one vector’s norm as grows: it falls toward the max norm of Section 4.

Section 3

The L2 norm and its square

The case is the Euclidean norm, the crow distance. Its subscript is usually dropped: . For it is .

Machine learning often uses the squared L2 norm, a plain sum of squares:

The squared norm is almost flat near the origin. For one entry , the squared norm is and the other two equal . Slide towards zero and compare the slopes.

How fast the size grows near zero

tangent lines at the chosen entry
Challenge +8 XP

Slide to a nonzero value below 0.05 in size, where is almost flat but still climbs at slope 1.

The slope of is , so near zero the square hardly notices a small entry. The slope of is at every nonzero point.

The squared norm is more convenient to work with, mathematically and computationally, because its derivatives depend only on the matching entry. The book gives this reason on page 39. The derivative with respect to is . The plain norm’s derivative needs the whole vector, through the denominator. This derivation is added here and is not printed in the book.

Going deeper

This derivation is added here and is not printed in the book. The L2 norm is . The chain rule gives . Dividing by the same length is what couples the entries: moving one entry changes every partial derivative of the plain norm.

Test that claim. You will nudge entry 1 and watch entries 2 and 3.

Derivatives of the squared and the plain L2 norm

Predict: if only entry 1 changes, which derivatives of entries 2 and 3 will change?

Section 4

The L1 norm, the nonzero count and the max norm

When the difference between exactly zero and slightly nonzero matters, the book uses the L1 norm, the sum of absolute values:

(2.31)
Click any coloured symbol to see what it means.

It grows at the same rate everywhere. Move an entry away from 0 by and the L1 norm rises by exactly . The squared L2 norm rises by only .

Going deeper: counting nonzero entries

You may also want to count the nonzero entries. Some call that the “L0 norm”. The book says that name is wrong: scaling by leaves the count unchanged, but a norm must scale by . The L1 norm is often used as a substitute. (The book gives no argument for that in this section.)

The last vector norm is the max norm, or L∞ norm, the entry of largest magnitude:

(2.32)
Click any coloured symbol to see what it means.

Build a vector of six entries by dragging the bars up and down. The table measures it four ways.

Four ways to measure one vector

entries of (drag the bars)entries of

Set the scale factor to 2: L1, L2 and max double, while the count of nonzero entries stays put. Then raise : entry 4 starts at exactly 0, so the count jumps from 3 to 4 for any positive , however small. L1 rises by , the square by only . Entry 5 sits at 0.05: it adds 0.05 to L1 but only 0.0025 to the squared norm.

Challenge +8 XP

Drag the bars to a vector whose L1 norm is 3 (within 0.1) but whose max norm is 1 (within 0.05): many small entries instead of one big one.

Section 5

The Frobenius norm of a matrix

A matrix needs a size too. In deep learning the usual choice is the Frobenius norm, which the book calls otherwise obscure: square every entry, add everything, take the root.

(2.33)
Click any coloured symbol to see what it means.

It is the L2 norm of the matrix unrolled into one long vector, like unrolling a carpet. Set the six numbers of a 2×3 matrix and compare both routes.

The Frobenius norm is the L2 norm of the unrolled matrix

Challenge +8 XP

Set the six numbers so the Frobenius norm is exactly 5.

Predict the norm of four ones

Predict first. Start from a 2×3 matrix of zeros and set four of its entries to 1, one at a time. What is the Frobenius norm at the end?

Four ones, what norm?

Both routes agree, because they do the same arithmetic in a different order. Because entries are squared first, one big entry dominates. Try a single 3 and five zeros: the norm is 3.

Section 6

The dot product and the angle

This section connects the norm to the dot product of Section 2.3. With the angle between and :

(2.34)
Click any coloured symbol to see what it means.

Read it as alignment. Same direction: and the dot product is the product of the lengths. A right angle: and the dot product vanishes. Opposite: the negative of the product. Guess the sign before you see it.

Predict the sign of the dot product

Now drag both tips and watch two routes to the same number: the entries, and the lengths with an angle.

Two routes to the dot product

angle
Challenge +8 XP

Drag the vectors so they are perpendicular (within about a degree): the dot product drops to zero.

The two numbers agree for every position. Positive within 90 degrees, zero at 90, negative beyond: the dot product follows the cosine. Orthogonal vectors have dot product 0 whatever their lengths.

Going deeper

The book states Equation 2.34 without proof. This derivation is added here and is not printed in the book. Write the squared length of the third side of the triangle two ways: with the law of cosines (first block) and by expanding the dot product (second block).

The terms and cancel on both sides, leaving . That is Equation 2.34.

What to take away
  • A norm maps a vector to a non-negative size. It is zero only for the zero vector, obeys the triangle inequality and scales with .
  • The Lp norm (Eq. 2.30, ) covers the L1 sum of absolute values (Eq. 2.31), the Euclidean L2 norm and, as grows, the max norm (Eq. 2.32).
  • The squared L2 norm has simple derivatives but is flat near the origin. The L1 norm grows at the same rate everywhere and so separates zero from small nonzero entries.
  • The count of nonzero entries is not a norm, since scaling does not change it. The L1 norm is often used in its place.
  • The Frobenius norm (Eq. 2.33) is the L2 norm of the matrix entries. The dot product equals the product of the two L2 norms and the cosine of the angle (Eq. 2.34).
Final quiz

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.5 (pages 39–40), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.