Architecture Design

Layers as a chain

The architecture of a network is its wiring: how many layers it has, how wide each layer is, and which units feed which. Think of an assembly line. Each station takes the part from the station before it, does one job to it, and passes it on. The finished product depends on every station.

The simplest architecture is a chain. The first layer takes the input and produces a list of hidden values . The second layer takes that list and produces the next one. The first two layers are

and the same pattern repeats for every further layer. Layer has a weight grid , a bias vector and an activation function that is applied to each entry separately.

Two numbers describe a chain. The depth is the number of hidden layers, which is how the demos below count it. The width is the number of units in a layer. The book names these two as the main decisions for a chain. It adds that a single hidden layer is enough to fit the training set, while deeper networks often need far fewer units and parameters but can be harder to optimize. The ideal architecture is found by experiment, guided by the error on a validation set.

Every weight and bias is a parameter, so depth and width also decide how many parameters there are. Predict the count before you use the builder.

Each connection between two adjacent layers is one weight, and each unit in a layer has one bias. The count in the readout follows from that rule. The counting rule is written here for this page and is not printed in the book.

Enough hidden units

A linear model can represent only linear functions. It is easy to train, because many of its losses are convex, but real problems need curves. So the natural question is whether a network with hidden units can represent any curve at all. The book answers yes, under a condition. The universal approximation theorem (Hornik et al., 1989; Cybenko, 1989) says that a feedforward network with a linear output and at least one hidden layer with a squashing activation can approximate any Borel measurable function from one finite-dimensional space to another. The error can be made as small as you like, provided the network has enough hidden units.

The book adds three points. The derivatives of the network can also approximate the derivatives of the function (Hornik et al., 1990). The book does not develop Borel measurability. For its purposes it is enough that every continuous function on a closed, bounded set is Borel measurable, so that is the case you meet in practice. The result also extends to a wider class of activation functions, including the rectifier from Section 6.3 (Leshno et al., 1993).

The theorem is easier to trust after you see one construction. The demo below uses one hidden layer of rectifiers to match a wiggly target curve. Each hidden unit places a bend, called a knot, at one point along the input. The network output is a straight line between knots. This construction is added here and is not printed in the book.

Here is the target curve, is the position of the knot of unit , is how much the slope changes at that knot, and is a constant. Each hidden unit computes one rectifier, and the output adds them up with weights .

The network curve has one straight piece for each hidden unit, so the curve becomes smoother as you add units. The gap shrinks quickly at first and then more slowly. Move the slider up from 2 and watch the gap in the lower plot. Then find the point where the largest gap falls below 0.05.

Enough units is not a guarantee

The theorem says that a large enough network exists. It does not say how large. Barron (1993) gives bounds on the size of a single layer for a broad class of functions, but in the worst case the number of hidden units can grow exponentially. Each input pattern might need its own unit.

The count below makes this concrete. A binary function takes a vector of bits and returns one bit. There are such functions, and choosing one needs bits, one for each input pattern. That is the book’s point: the choice of one function has about degrees of freedom. Ask yourself how many bits a function of 5 inputs needs.

For up to four inputs the demo draws the whole table, so you can click each output. For larger the table is too long to draw, and the readout gives the counts instead. The number of functions grows far faster than any list of examples could cover.

The book gives two separate ways in which learning can still fail. The optimizer may not find the parameters that represent the right function. The training procedure may also pick a function that fits the training set and does not generalize, which is overfitting from Chapter 5. The theorem speaks to neither failure.

Depth folds the input space

The book summarizes the point so far. A single hidden layer can represent any function, but that layer may be infeasibly large, and it may fail to learn and generalize correctly. In many circumstances a deeper model can use fewer units and reduce the generalization error. Sections 4 and 5 show where the advantage of depth comes from.

The book describes a different kind of result, from Montufar et al. (2014). Consider a network whose hidden units apply the absolute value. Each such unit folds the input space like a sheet of paper. Two points that are mirror images across the fold get the same output. A function computed on top of the fold is a mirror of a simpler pattern. Stack folds on top of folds and the number of straight pieces grows very fast. Figure 6.5 of the book shows this picture.

A one-dimensional example makes the folding exact. On the interval from 0 to 1, the tent map is one minus the absolute value of 2 times the input minus 1. That is an absolute value fold. The same map can also be written with two rectifiers, as the second line shows, and applying it doubles the number of straight pieces each time. These two identities are added here and are not printed in the book.

Here is the input, between 0 and 1, and is one layer with two hidden units. Apply once and the curve has two straight pieces. Apply it twice and it has four. The demo lets you choose the number of layers.

Each layer adds two hidden units, and each one doubles the pieces. The number of pieces after layers is . The book’s equation 6.42 states the same idea for many inputs, which is covered in Section 5.

Bounds on the number of pieces

For general inputs the book gives an upper count on the number of linear regions of a deep rectifier network. With inputs, depth and units in each hidden layer, the count is of order

The symbol means that constant factors are dropped, so only the scale is meaningful. The count grows exponentially in the depth , because appears in the exponent. For a maxout network with filters per unit, the book gives the order

Again the depth sits in the exponent. The book notes that some functions a deep rectifier network represents may need an exponential number of hidden units in a shallow network. It also says there is no guarantee that the functions of real tasks have this property.

Results of this kind were first proved for circuits of logic gates (Håstad, 1986) and later extended to networks with continuous-valued activations (Maass, 1992). The results of Leshno et al. (1993) say nothing about depth or efficiency. They only show that a sufficiently wide rectifier network can represent any function.

The book also gives a reason from the prior. The choice of a deep model states a belief that the target function is a composition of simpler functions. One reading is that the learner should discover factors of variation that build on each other. Another reading treats the network as a program with steps, where each step uses the output of the one before. Empirically, depth often improves generalization across many tasks, as the book’s citations show.

Explore the scale with the demo. The horizontal axis is the depth, and the vertical axis is the order of the count on a logarithmic scale, so a straight line means exponential growth.

With one input and two units per layer, the order in equation 6.42 is exactly . This is the tent construction from Section 4, so the two results agree in that case. Set to 1 and to 2 to check it.

Other architectural considerations

The layers need not form a plain chain, though a chain is the most common choice. Many architectures keep the main chain and add extra links. A skip connection sends the output of layer directly to layer or further on. The book says these connections make it easier for the gradient to flow from the output back to the layers near the input. The demo counts the shortest route through the network, which is one measure of how far a signal must travel. That count is added here and is not printed in the book.

The second design choice is how the layers connect. In the default layer every input unit feeds every output unit. A sparse layer lets each output see only a few inputs. Convolutional networks, covered in Chapter 9, use this idea, and the book says such patterns work well for vision. Recurrent networks for sequences, in Chapter 10, have their own architectural considerations. The next demo counts the parameters of a sparse layer in which each output unit connects to neighboring inputs. Here is a count of connections per output, and the layer wraps around at its ends.

Full connection needs weights, and a band with connections per output needs weights. Both need one bias per unit. The band layer is much smaller and needs less computation, but it can only mix nearby inputs. Whether that is a good trade depends on the problem, and the book says architecture choices of this kind are problem-dependent.

Evidence and the takeaway

The book supports its depth argument with experiments. Figure 6.6 reports test accuracy for transcribing multi-digit numbers from photographs of addresses. In the experiment accuracy rises as the network gets deeper (Goodfellow et al., 2014d). Figure 6.7 shows a control: widening the layers of a convolutional network without adding depth does not give the same gain. The caption reports that shallow models overfit at around 20 million parameters, while deep models can benefit from more than 60 million.

This page does not rebuild those experiments, and it does not reproduce their numbers. The demo below shows the mechanism behind the claim instead. It compares a deep tent network with a one-layer network that has the same number of hidden units. The one-layer curve is one simple construction, chosen for clarity, and it is not the best possible one-layer network.

Compare the two curves at depth 5 and depth 6. The deep curve gains pieces faster than the one-layer curve, which is the point of Section 5. A one-layer network with H units has at most H plus 1 pieces, because each unit adds at most one bend. That bound is added here and is not printed in the book. Keep in mind that the one-layer curve here is one choice. The gap it shows belongs to this construction only, and says nothing about the best one-layer network with the same units.

• A chain of layers computes each layer from the one before, with equations 6.40 and 6.41 as the pattern. Depth and width are the main design choices.
• One hidden layer is enough to represent a function with enough units, but the theorem does not say how many. Worst-case counts grow exponentially.
• Rectifier networks gain pieces exponentially with depth, as equations 6.42 and 6.43 describe. Depth is a prior about composed functions, not a guarantee.
• Skip connections shorten the route a signal must travel, and sparse connections cut the parameter count. Both are problem-dependent design tools.
• The book’s depth evidence comes from experiments, which this page does not reproduce.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 6.4 (pages 197–203), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.