Multiplying Matrices and Vectors

One operation, many weighted sums

A neural network layer turns one list of numbers into another, one weighted sum at a time.

Picture a stack of recipe cards. Card says how much of each ingredient goes into dish . The matrix is that whole stack.

Recap: has rows and columns, and (the transpose) swaps rows and columns.

A positive weight on a positive input pushes right. A negative weight or a negative input pushes left. Sum and difference adds all three inputs in row 1 and compares the first input with the third in row 2.

Your challenge answer shows something odd: a non-zero input can produce a zero output. Keep that in mind for the module on span and rank.

The matrix product and its shapes

The most important operation on two matrices is their product. The product of and is a third matrix , written by placing them side by side:

Think of it as a handshake between shapes. must have as many columns as has rows. With of shape and of shape , the result is . The inner numbers must agree and disappear.

Entry by entry, the product is defined by

To build , take row of and column of . Pair their -th entries, multiply each pair, and add up all products. The index runs along the shared size, which is why that size has to match.

Click any cell to choose which entry of to build. Then type the entry yourself before reading it.

With the default sizes, 2 by 3 times 3 by 2 gives 2 by 2. Raise “Rows of B” to 4 and the product disappears: a row of has 3 numbers and a column of has 4, so they cannot be paired.

The element-wise product

The standard product is not the matrix of products of matching entries. That second operation is the element-wise product, or Hadamard product, written . Both matrices need the same shape:

Switch the explorer to its second tab and compare. There is no sum over at all. With , the matrix product has every entry equal to 2 (each entry adds two products), and the element-wise product is all ones.

The dot product

Two vectors and of the same length have a dot product. The book defines it as the matrix product : a row (shape ) times a column (shape ) is a 1 by 1 matrix, which is one number. Equation 2.5 with those shapes gives the formula:

This also explains the book’s remark that in a matrix product is the dot product of row of with column of . The recipe demo was a stack of dot products.

A dot product measures agreement. Entries with matching signs add to the total, and opposite signs subtract. Drag the two arrow tips.

Arrows pointing roughly the same way give a positive total. A right angle gives zero. Arrows pointing apart give a negative total. Equation 2.34, in the section on norms, makes the link to the angle exact.

The second bar chain multiplies the same pairs in the other order, , and ends at the same place. Every product equals , since ordinary multiplication ignores order. That is the first rule of the next section.

Rules of the product and the transpose

The matrix product obeys two friendly rules. It is distributive and associative:

Distributive means a product can be spread over a sum. Associative means the brackets can go anywhere in a chain, so is unambiguous. The shapes still have to fit at every step.

One rule from school arithmetic does not survive. Matrix multiplication is not commutative: does not always hold, and for some shapes does not even exist. A pair of vectors is the exception, and the book records it:

The bench tests each rule on random matrices. It is a check, not a proof.

Distributivity and associativity hold for every draw. The third tab is where things fail: for 2 by 2 matrices both orders exist and usually differ, and for rectangular shapes cannot be formed. A few pairs do commute, which is why guessing is a game.

Order matters: a picture

Order matters because a matrix acts, and acting happens in time: socks, then shoes, is not shoes, then socks. By Equation 2.7,, so means “act with first, then ”.

Here turns the plane and stretches the horizontal direction. Drag on either picture to turn, and use the slider to stretch.

The two results agree only in special settings: no rotation, no stretch, or a half turn (which sends every point through the origin and so commutes with any stretch). Everywhere else the orders disagree.

The transpose of a product

The transpose of a product has a simple form, with a twist: the order reverses.

The shapes demand the twist. If is and is , then is . The product multiplies by and lands on the same shape. The other order, , only fits if . Try the fourth tab with rectangular shapes.

Equation 2.9 also explains Equation 2.8. The value of is one number, and one number is its own transpose. Transposing the product reverses the factors and leaves the value alone:

The book adds that it does not list all useful properties of the matrix product, since linear algebra is not its focus, and that many more exist.

Systems of linear equations

The notation is now rich enough to write a whole system of linear equations in one line:

Here and are known, and holds the unknowns. A system reverses the first demo: the output is fixed, and you hunt for the input.

Each row of , together with its entry of , is one constraint:

Each line is a dot product of a row with . Written out entry by entry, the system reads

There are equations, one per row, and unknowns, one per column.

With two unknowns, each equation is a line in the plane of the unknowns, and a solution lies on all lines at once. Drag the point around the plane.

Two independent rows give exactly one solution, where the lines cross. A third equation misses that crossing unless it passes through it. Parallel rows either coincide, giving a whole line of solutions, or never meet, giving none. The next sections of the book explain what decides which case you get.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.2 (pages 34–36), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.