Scalars, Vectors, Matrices and Tensors

Four kinds of arrays

Picture a spreadsheet. One cell holds one number. A column of cells is a list. A whole sheet has rows and columns. A workbook of sheets needs a third coordinate to find a number.

The book gives these four things names. The only difference between them is how many coordinates you need to point at one number.

• A scalar is one number, with no index. It is written in italics, lower case, like .
• A vector is an ordered list. One index names an element. It is bold lower case, like .
• A matrix is a grid. Two indices name an element. It is bold upper case, like .
• A tensor is a grid with any number of axes. Three indices name an element of a 3-axis tensor.

Switch the tabs, resize the array, and click a cell. The readout tells you the book’s name for the cell you picked and how many numbers the array stores.

Each added axis multiplies the count by its length. That is why deep learning libraries pack so much data into tensors. Test your feel for it.

Vectors as lists and as points

A vector is a list where position matters. The number at each position is found by its index: the first element of is , the second is , and so on.

If every element is a real number and there are of them, the vector lives in . That symbol is the set made by multiplying the real numbers by themselves times, called a Cartesian product. Written as a column:

A vector with lives in , which you write .

A vector is a point in space

The book suggests another picture: a vector names a point, and each element is the coordinate along one axis. Two elements give a point in a plane. Three give a point in ordinary space. Deep learning uses vectors with thousands of elements, and the idea does not change.

Drag the point. Then try to land it on the dashed ring.

The vector and the point are one object. The column of numbers is how a computer stores it.

Indexing a set of elements

Sometimes you want several elements at once. Name the wanted positions in a set and write it as a subscript. With , the symbol is the vector holding , and .

A minus sign takes the complement: holds everything not in , and holds everything except . Click cells to move them in and out of .

The two parts always split the vector with no overlap, so their lengths add up to .

Matrices and their indices

A matrix is a grid, so each element needs two indices: row first, column second. A real matrix with rows and columns lives in , and is its shape.

Think of seats in a cinema. is the top left seat and the bottom right one. The element’s name is italic, not bold.

A colon means “all of them”. is the whole row , and the whole column . Click a cell below and see both.

Indexing a matrix-valued expression

When the matrix is a whole expression, the subscript goes after it. The book’s example is : apply the function to , then pick the element. The order matters. The tabs show two functions. acts on every element. The transpose, taught in Section 5, moves them.

The numbers are fixed, so enlarging the matrix reveals more of the same entries. A row has elements and a column has . In the transpose tab, reads , and the matrix on the right changes shape.

Tensors

Sometimes two axes are not enough. An array on a regular grid with any number of axes is a tensor. The book writes it in a sans-serif capital, like , and names one element with . Scalars, vectors and matrices are tensors with zero, one and two axes.

The standard example is a color photo. A gray image is a matrix of brightnesses. A color image adds a third axis for the channel (red, green, blue). The element is the intensity of channel at the pixel in row , column .

The picture below is a tiny synthetic image with values from 0 to 9 per channel. Click any pixel or any number.

Fix and you get a matrix: one channel. Fix and and you get a vector of three numbers: one pixel’s color. A batch of images adds a fourth axis.

The transpose

Hold a sheet of paper up to a mirror placed along its main diagonal, the line from the upper left corner down and to the right. What you see is the transpose. The transpose of is :

Row , column of the result is what sat at row , column before. Rows become columns, so an matrix becomes . Diagonal elements stay put, and flipping twice restores the matrix.

This is the book’s Figure 2.1, a 3 by 2 matrix flipped into a 2 by 3 one. Press Flip or drag the fold slider. Click any element to follow it across the diagonal.

A vector counts as a matrix with one column, so its transpose has one row. That is why the book writes a vector inline as a row and adds : . A scalar is a matrix with one cell, so it is its own transpose. Try the three buttons.

Now build a matrix that is its own mirror image. Edit the numbers. The cells whose mirror partner disagrees turn orange.

A matrix equal to its own transpose is called symmetric. The book names these in Section 2.6.

Adding arrays and broadcasting

Two matrices of the same shape add cell by cell, like laying one transparent grid over another:

Different shapes have no sum. A scalar is allowed to scale and shift every element of a matrix:

The book then adds a shortcut common in deep learning. A matrix of shape plus a vector with elements is another matrix:

Imagine photocopied once for every row, then added. The book writes the shortcut so you need not build that copy. The implicit copying is called broadcasting. It needs one entry of per column. The demo draws the copies as faint rows.

Break it on purpose: in the first tab add a column to , and in the third tab change the length of away from . The sum stops existing. This is the book’s notation. How a given software library treats other shapes is a separate matter.

• A scalar is one number, a vector a list (one index), a matrix a grid (two indices), a tensor any number of axes. The count of stored numbers is the product of the axis lengths.
• A vector in has real elements and is a point. picks the elements in and picks the rest.
• A matrix in has rows and columns. is an element, a row, a column. For an expression the subscript goes after it: .
• The transpose mirrors across the main diagonal: . A column becomes a row, and a scalar is its own transpose.
• Equal shapes add cell by cell, scalars act on every cell, and a vector with one entry per column is added to every row by broadcasting.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 2.1 (pages 31–34), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.