Explicit Memory and Soft Reads

Two kinds of knowledge

The book makes a distinction that is easy to miss. Some knowledge is implicit: how to walk, or how a dog looks different from a cat. Other knowledge is explicit, such as a commonsense fact or a detail you need for today, like a meeting time. Neural networks store implicit knowledge well. They struggle with explicit facts. Stochastic gradient descent needs many presentations of the same input before it is stored in the weights, and even then it is not stored especially precisely.

The book cites Graves et al. (2014b), who hypothesize that networks lack a working memory like the one people use to hold and manipulate information for a goal. Explicit memory components would let a system store and retrieve specific facts rapidly and on purpose. One design is a network with a set of memory cells that the task network can read and write, as in memory networks (Weston et al., 2014) and the neural Turing machine (Graves et al., 2014b). The soft read in section 3 follows the neural Turing machine in outline. The book also notes that explicit memory seems to let models learn tasks that ordinary RNNs and LSTMs cannot, perhaps because information and gradients can travel over very long durations when cell content is copied rather than forgotten. The sections below follow the order of the book.

Predict first. Which option can keep the fact "the meeting is at 3:00 PM in room 141" exactly, after one presentation?

Pick an answer. The next section shows how a fact is stored by gradient descent.

Storing facts by gradient descent

To store a fact in weights, the network must be shown the same input many times. Each presentation moves the weights a little toward the target, so the fact fades in slowly. The demo below uses one weight as a stand-in for a fact. This toy is added here and is not printed in the book.

Let the target be 1 and the starting value be 0, so the first error is = 1. Each presentation is one gradient step of size on the squared error, and the error shrinks by the same factor each time. After presentations the error is:

(This formula is added here and is not printed in the book. It follows from one gradient step per presentation.) In this toy, a one-shot write into an explicit cell has error 0 after one step, because the cell holds the exact content.

With a small step size the error falls slowly, and many presentations are needed. A larger step size is faster. This toy does not model interference between facts, so it shows speed but not the full cost of learning by gradient steps.

Memory cells and soft reads

Figure 10.18 in the book shows a task network that controls a memory. It decides where to read and where to write. The book notes that functions which produce exact, integer addresses are difficult to optimize. Its answer is to read many cells at once. The coefficients are chosen to focus on a small number of cells, for example with a softmax. This module uses the following rule, which the book describes in words but does not print:

The read is a weighted average of the cells, so the value read is:

The sharpness controls the focus. At every cell gets the same weight. As grows, the largest score takes almost all the weight, and the read approaches one cell.

The scores favor the target cell and its neighbors. A soft read therefore blends the target with nearby cells, and the blend is close to the target only when the focus is sharp enough.

Content and location addressing

The book separates two ways to address a cell. Location-based addressing says "read slot 347". Content-based addressing says "read the cell whose content matches this pattern". The book gives the example of recalling a song from a few words of its chorus. Content addressing is more useful when the stored objects are large, and location addressing works well even for small cells. The book gives two reasons to make memory cells vectors rather than single numbers. Reading a vector offsets part of the cost of producing many coefficients, and vectors allow content-based addressing.

In this module the score for content addressing is the cosine between the query and each cell, which is added here and is not printed in the book:

Predict first. Which cell does content addressing favor when the query is , and which does location addressing read when the slot is 3?

Pick an answer, then move the query and the slot to test it.

Content addressing reads whatever matches the query. Location addressing reads the slot you name, whatever the query says. The two are different tools, and the book's view is that content addressing is the one that lets a network find a stored object from part of it.

Hard reads and attention

The book describes an alternative to a soft read. The weights can be read as probabilities, and the network can read one cell, chosen at random with those probabilities (Zaremba and Sutskever, 2015). This hard, stochastic read makes a discrete choice. The book notes that training such models is harder than training models that make soft decisions, and that the soft version is the one that allows back-propagation through the read.

The addressing mechanism is the same attention mechanism used in machine translation (Bahdanau et al., 2015). Attention was first introduced for handwriting generation (Graves, 2013), where the focus could move only forward through the sequence. In machine translation and memory networks, the focus can move to a completely different place at each step.

A hard read picks one cell, so the value read is exactly that cell's content. The probability of picking the target is shown in the readout. When it is high, the hard read almost always returns the target.

The book closes this chapter with a remark on recurrent networks. They provide a way to extend deep learning to sequential data. The book calls them the last major tool in our deep learning toolbox, and the next part of the book turns to how to choose and use these tools on real tasks.

• Neural networks store implicit knowledge well, and explicit facts need many presentations before they are stored in the weights.
• An explicit memory keeps a set of cells that a task network can read and write, so specific facts can be stored and retrieved.
• A soft read takes a weighted average of many cells, so gradients can flow through it, and the sharpness controls the focus.
• Content addressing matches a query to the content of each cell, while location addressing reads a named slot.
• A hard, stochastic read picks one cell at random, which is harder to train than a soft read.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 10.12 (pages 416–420), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.