Deep Learning and Its Roots

Machines that learn from experience

People have dreamed of thinking machines for thousands of years. Today artificial intelligence (AI) reads speech and images, helps doctors and automates routine work. This lesson covers the book’s opening chapter: why learning beat hand-written rules, and what the word deep means.

The easy kind: rules fit on a card

Pick a piece, then click a square. The colored squares are every move it has on an empty board. The whole move generator is three lines.

This is a toy of movement only. Captures and check add a handful of lines, not a new kind of difficulty. Think of a recipe and a skill: a recipe fits on a card, riding a bicycle does not.

The hard kind: a rule for a wheel

Now the task the book calls hard: find cars in photographs. A car has wheels, so you want a rule for “wheel”. Shadows, glare and fenders change how a wheel looks.

Rule A asks “is the ring darker than a fixed number?”. Night wants a small number and strong sun wants a big one, so no setting fits all ten. Rule B compares the ring with its own surroundings, which survives changes of light, and it still fails on something.

Every repair invites the next exception, and nobody can foresee them all. That is why the book asks for a different plan: let the machine find the pattern.

From hard-coded knowledge to machine learning

You could not write the wheel rule by hand. So who should write the knowledge down: a person, or the machine?

Hard-coded knowledge

Several projects typed knowledge about the world into a computer in a formal language and let it reason with logic. This is the knowledge base approach, and none of these projects led to a major success. The best known is Cyc (Lenat and Guha, 1989): an inference engine plus a database of statements in a language called CycL, entered by human supervisors.

Letting the machine find the pattern

The way out is for the system to acquire its own knowledge by extracting patterns from raw data. This ability is machine learning. It lets computers handle problems that need real-world knowledge, and decisions that look subjective.

The book names two simple examples. Logistic regression can decide whether to recommend a cesarean delivery (Mor-Yosef et al., 1990). Naive Bayes can separate legitimate e-mail from spam. Figure 1.3 draws the model as a graph, and it is written as:

The second line expands the dot product for two inputs (added here, not printed in the book).

The input is a list of numbers, holds one weight (a volume knob) per input number, and squashes the total into a range between 0 and 1. The Figure 1.3 caption names as the logistic sigmoid but gives no explicit formula for it, so here is one (added here):

The dependence on features

Such a model sees only what it is handed. For the cesarean decision it never looks at the patient. The doctor feeds in facts, such as whether a uterine scar is present. Each fact is a feature, and the set of facts is the representation.

The model learns how each feature correlates with the outcome. It cannot change how the features are defined. Hand it an MRI scan and it fails, because a single pixel says almost nothing about complications.

Gradient descent does what your hands did, only automatically: it nudges every knob to reduce the error. Untick feature 1 and train again. The accuracy falls, because the model cannot invent a feature you did not hand over. That is the limit the book describes, and the next section shows why it matters.

Representation and the Cost of Description

Draw one straight line to split the teal dots from the orange ones. If that looks impossible, change how you describe the dots, not the line.

Searching a collection can be exponentially faster when it is indexed well. Arabic numerals make division easy, Roman numerals hard. Same problem, different description, different difficulty.

Figure 1.1 of the book shows this with a scatter plot. Here is a seeded data set: class A on a ring of radius about 1, class B on a ring of radius about 2.2.

Now describe each point by how far it is from the center and which way it points, like a ship’s distance and bearing from a lighthouse (standard geometry, added here):

Here is the distance and the angle of the point . Drag the two round handles to place a line. Then switch the representation.

In the Cartesian tab the classes are two nested rings, and every line cuts both. In the polar tab they become two vertical bands, and one vertical line splits them. Only the description changed, and the learner stayed as simple.

When the right features are designed, a simple learner works for many tasks. The book’s sound example: an estimate of the size of a speaker’s vocal tract hints whether the speaker is a man, a woman or a child. For a wheel, nobody knows which feature to extract.

Representation learning

What if the machine designed its own features, instead of you? That is representation learning: learn the mapping from representation to output and the representation itself.

Learned representations often beat hand-made ones, and a system can adapt to a new task with little human effort. A learning algorithm can find a good feature set for a simple task in minutes, or for a complex task in hours to months. Hand design for a complex task can take a whole community of researchers decades.

The autoencoder

The textbook example is the autoencoder, two functions back to back. The encoder turns the input into a code . The decoder turns the code back into the shape of the input. Chapter 1 of the book does not print these formulas, so they are added here:

Picture describing a face in one sentence, and a friend drawing it from that sentence. The code is the sentence. The reconstruction error measures what the code loses, and its form is added here:

Our toy autoencoder squeezes 2D points to one number. Aim the code line yourself: the code of a point is its position along the line, and the reconstruction is where that position lands in the plane. Drag the handle on the line, or use the slider, to set its direction .

The loss is lowest when the line runs along the long direction of the cloud. What remains is the spread across the line, which one number cannot keep. (This fact about the best linear code is added here, and the page checks it against the exact answer.)

Now let the machine do it. Four numbers (two encoder weights, two decoder weights) start small and random. Gradient descent on the loss above adjusts them.

It finds the same direction you did. Nobody told it what a good feature is. It found one because keeping the long direction preserves the most information.

Factors of variation

The goal of any feature design is usually to separate the factors of variation behind the data. A factor is a separate source of influence (not a multiplicative factor). Factors are often not observed directly. They may be objects or forces in the physical world, or constructs in the mind that give simple explanations for the observed data.

For speech: age, sex, accent and the words spoken. For a car photo: position, color, and the angle and brightness of the sun.

The difficulty is that many factors touch every single piece of data. A red car is nearly black at night. We must disentangle the factors, keeping those we care about and discarding the rest.

A code with one number per factor, like the three numbers under the picture, would be easy to use. The job of a good learned representation is turning pixels into those numbers.

The book is plain about the catch. Factors such as a speaker’s accent can be identified only with nearly human-level understanding. When a representation is almost as hard to get as the original answer, representation learning does not at first glance help. Depth is the answer.

Depth: concepts from simpler concepts

Build a hard idea out of easy ones, like a sandwich of layers: dots, then edges, then corners, then shapes. That is the deep learning answer.

It introduces representations expressed in terms of simpler representations. The book’s Figure 1.2 builds the concept “image of a person” from corners and contours, which are built from edges.

The classic deep model is the feedforward deep network, or multilayer perceptron (MLP): a function made by composing many simpler functions. Each one gives a new representation of the input. The book describes this in words, so the display below is added here and is not printed in the book:

Read it inside out, like an assembly line. hands a first description to , which hands a more abstract one to . The superscripts only count layers, and the three layers here are a choice made for this page.

A pipeline by hand

Here is a hand-made toy with the same shape of computation. A trained network learns its own features, and these rules were written by hand for the picture.

Layer 1 marks an edge piece wherever two neighboring pixels differ. Layer 2 looks only at edge pieces and marks a corner where exactly two meet at a right angle. Layer 3 counts corners. The word “corner” never appears in the pixel layer.

Composition makes complexity cheap

Why would stacking help? (An illustration added here, not from the book.) The tent function rises from 0 to 1 and falls back. Feed its output into itself, and each use is one more layer.

With layers there are peaks and straight pieces, yet the whole thing is copies of one two-piece rule. Written out by hand, every piece would grow exponentially. A cheap way to describe a wiggly function is to reuse simple steps.

Depth as a multi-step program

The book gives a second view: depth lets the computer learn a multi-step program. Each layer is the state of the computer’s memory after another batch of instructions, and later steps can use earlier results.

Two ways to measure depth

How deep is a model? Ask two people and you may get two answers. Both can be right.

The book gives two views. View 1 counts the sequential instructions needed to evaluate the model: the longest path through a flowchart. View 2, used by deep probabilistic models, counts the depth of the graph that relates concepts.

View 1: the longest path through the computation

A flowchart is a directed graph: boxes joined by arrows. Inputs have depth 0, and every other box is one step deeper than its deepest input. The depth of box is written , and an arrow from box into box is written :

The same function gets a different depth depending on which functions count as one step. In Figure 1.3 the logistic regression has depth 3 with multiply, add and sigmoid as steps, and depth 1 if logistic regression itself is one step.

The depth of the model is the depth of its output box. (The recursion is a precise version of the book’s “length of the longest path” and is added here.) The middle language (a dot product as one step) is an addition made on this page. None of the three numbers is wrong, because they measure different things.

View 2: the depth of the graph of concepts

Deep probabilistic models count concepts instead of computations. The computation for each concept can be far deeper than the concept graph, because simple concepts are refined using more complex ones.

The book’s example: a system looks at a face with one eye in shadow. At first it sees only one eye. After it detects the face, it infers that a second eye is probably there. The concept graph has two layers, eyes and faces. The computation graph has layers if each concept’s estimate is refined times given the other:

The toy makes the loop explicit. Two eyes and a face are three beliefs in log-odds form (positive means “probably present”). Each round, the face belief is computed from the eyes, then each eye is re-judged from its own evidence plus a push from the face. The coupling numbers are chosen for illustration and mean nothing beyond the picture.

At the shadowed eye is judged from its own pixels. Each round, the clear eye supports the face and the face supports the hidden eye. The concept graph never gets deeper, yet the computation has steps. Set the coupling to 0 and refining does nothing.

No single depth is correct, and no consensus exists on how deep a model must be to count as deep. The book’s safe statement is that deep learning studies more composition of learned functions or learned concepts than traditional machine learning does.

AI, machine learning and deep learning

Four nested boxes hold the whole chapter. See where each kind of system sits, and what changes in its flowchart.

Figure 1.4 is a Venn diagram: deep learning inside representation learning, inside machine learning, inside AI. Figure 1.5 shows flowcharts of how the parts of an AI system relate, with shaded boxes for the components that learn from data. Pick an example.

Deep learning is an approach to AI and a kind of machine learning, which lets computers improve with experience and data. The authors consider machine learning the only viable way to build AI that works in complicated real-world settings. Deep learning builds a nested hierarchy of concepts, each defined through simpler ones.

Go left to right through the four examples. A rule-based system learns nothing, classic machine learning learns the mapping, and representation learning learns the features too. Deep learning stacks more learned layers.

Intended Readers and Reading Paths

The book has two target groups. University students, undergraduate or graduate, learning machine learning, including those beginning a career in deep learning or AI research. And software engineers with no machine learning or statistics background who want to acquire one quickly and use deep learning in a product or platform.

The book has three parts. Part I covers mathematical tools and machine learning basics. Part II describes the established deep learning algorithms. Part III covers speculative ideas for future research.

The book assumes a computer science background: programming, basic computational performance, complexity theory, introductory calculus and some graph theory. Readers should skip the parts that do not suit their background or goal, and the arrows in Figure 1.6 show what depends on what.

Three waves of neural networks

The book states three trends. Deep learning has grown more useful as training data has grown, its models have grown as hardware improved, and it solves harder applications with rising accuracy. The book’s evidence for these lies beyond page 14.

Deep learning dates back to the 1940s and has gone by many names. The book counts three waves: cybernetics in the 1940s to 1960s, connectionism in the 1980s and 1990s, and the current resurgence under the name deep learning, beginning around 2006.

The name changed with the fashion. The first models were meant as computational models of biological learning, which is why the field was called artificial neural networks. The modern term goes further: it appeals to learning multiple levels of composition, a principle that also applies in frameworks that are not inspired by the brain.

• Rule-writing fails on tasks that are easy for people but hard to describe (wheels, faces, speech), because the exceptions never end.
• Machine learning extracts its own knowledge from data, but simple learners depend on the features they are handed.
• Representation learning learns the features too. Its standard example is the autoencoder, and its goal is to separate the factors of variation.
• Deep learning composes simpler representations into complex ones. There is no single correct depth, because it depends on which steps count.
• Deep learning began in the 1940s under other names. The current wave started around 2006.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 1.0 (pages 1–14), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.