Hidden Units

Hidden units as bends

A hidden unit takes the numbers from the layer below and returns one number. It works in two steps. First it forms a weighted sum of its inputs and adds a bias. That number is the pre-activation, . Then it passes the pre-activation through a fixed function, the bend, and the result is the unit’s output, . Training adjusts the weights and biases. The designer chooses the bend, and this page is about that choice.

Think of the unit as a valve on a water pipe. The pressure coming up the pipe is the pre-activation, and the flow out of the valve is the output. A straight pipe passes the pressure through unchanged. A one-way valve lets water through only when the pressure is positive. A saturating valve opens fully at high pressure and closes fully at low pressure, so extra pressure stops adding flow. The shape of the valve is the bend.

The book makes three points about the choice. No definitive theory yet picks the bend for a given problem. Rectified linear units are an excellent default. In practice the choice is trial and error: train a network with one kind of unit, then check the result on a validation set.

Predict first. Which bend has a slope of exactly zero over half of its domain?

Choose an answer.

The linear unit is a straight line, with slope 1 everywhere. The rectified unit copies the line where it is positive and is flat where it is negative. The sigmoid and tanh squeeze the whole line into a band, and both flatten at the ends. The cosine repeats itself, so one output value can come from several inputs. The next section looks closely at the corner of the rectified unit.

Derivatives at the kink

Most bends are smooth, but some have a corner. The rectified unit bends at zero: its slope is 0 on the left and 1 on the right. Gradient descent needs one slope at each point, so the corner needs a rule.

Two definitions make the rule precise. The left derivative is the slope of the bend immediately to the left of a point, and the right derivative is the slope immediately to the right. The bend has a derivative at a point only when the two agree. These limits are added here for clarity. The book states the same idea in words. The functions used in neural networks usually have both one-sided derivatives defined.

The book’s example is the rectified unit at zero. Its left derivative is 0 and its right derivative is 1, so no single slope exists there. The book says software usually returns one of the two one-sided values instead of raising an error. It gives a reason: when a unit is asked for its value at zero, the underlying value was very unlikely to be exactly zero, and was more likely a small value that rounded to zero. The book also says training does not normally reach a point where the gradient is exactly zero, so a minimum at a corner is acceptable.

Move the slider to the left to shrink the step. The picture always stays zoomed to the corner, and the two chords keep their slopes. The right chord is the right derivative and the left chord is the left derivative. A single number for the slope at zero does not exist. The average of the two chords is a central difference, which is neither one-sided value. This average is added here as an illustration.

Now the general form of a hidden unit. Unless the text says otherwise, a hidden unit takes a vector , forms the affine transformation, a weighted sum plus a bias, , and applies the bend element by element. Only the bend changes from one kind of unit to the next.

Here has one column for each unit and one row for each input, so is the matrix that maps the inputs to the pre-activations. The corner of the rectified unit is harmless in practice, the book says, because the bend is non-differentiable at only a few points.

Rectified linear units

The rectified linear unit uses the bend below. It keeps positive pre-activations and sets negative ones to zero.

Rectified units are typically used on top of an affine transformation, so each layer of them computes

The book’s reason for liking this bend comes from its slope. On its active side the unit is a plain line, so the slope is 1 and never shrinks. Wherever the unit is on, the gradient is large and consistent. The second derivative is zero almost everywhere, so the gradient direction stays useful. The unit is easy to optimize because it behaves like a linear unit.

The book suggests starting the biases at a small positive value, such as 0.1. The idea is that the unit is then likely to be on for most training inputs, so gradient passes through it. That outcome depends on the spread of the pre-activation as well as the bias. The demo below lets you change both. The dependence on the spread is added here as an analysis and is not printed in the book.

Assume the pre-activation is roughly normal with mean and spread . The chance that the unit is on is then , where is the normal distribution function. With the spread at 1, a bias of 0.1 keeps the unit on for about 54% of inputs. With the spread at 0.1, the same bias keeps it on for about 84%. A small bias works best when the spread is small.

The weakness is the other side. For an input on which the unit is off, the gradient through it is zero, so that example cannot change the unit’s weights. The book says rectified units cannot learn by gradient methods on examples where their activation is zero. A unit that is off for nearly every input is called dead in practice. The book does not use that word here, so the term is added.

Push the bias below −2 with the spread at 1. About 98% of inputs are then off, so almost nothing trains that unit.

Generalizations of rectified linear units

The generalizations keep the positive side of the rectified unit and give the negative side a slope. The book writes this with one slope for each unit, and an index for the unit:

Here is the slope on the negative side. With the unit is the plain rectified unit. The book names three choices.

• Absolute value rectification sets , which gives . The book uses it for object recognition from images, where features should not change when the illumination reverses polarity.
• Leaky ReLU fixes at a small value such as 0.01.
• Parametric ReLU, or PReLU, treats as a parameter that training learns.

A non-zero gives the unit a slope on its negative side, so gradient reaches the unit everywhere. The book says these generalizations usually perform about as well as the rectified unit and occasionally better.

Two points follow from the demo. Below zero the output is , so the output and the slope on that side both scale with . When is zero, the unit gives no gradient for any input in .

The book states the principle behind all of these units: a model is easier to optimize when its behavior is closer to linear. The same principle appears in recurrent networks, where the LSTM passes information through time by summation, as section 10.10 of the book describes.

Maxout units

A maxout unit does not bend each number on its own. It splits the pre-activations into groups of values and outputs the largest value in each group. The group for output uses the positions listed in the book’s definition of :

Each output is the largest of straight lines, one line per filter. The maximum of several lines is a convex function made of straight pieces. A convex function curves upward: every straight segment joining two points of its graph lies on or above the graph. Each piece is a stretch of the input where one line is the largest. The book says a maxout unit learns a piecewise linear, convex function with up to pieces. With large enough , it can approximate any convex function with arbitrary fidelity, so the unit learns the activation function itself rather than only the relationship between units. The demo shows how the pieces change as you move the intercepts.

The slopes of the four lines are fixed at 1.6, −0.8, 0.5 and −1.4, so only the intercepts move. Line 1 is the steepest rising line, and it becomes the winner in its group only when its intercept is high enough.

A maxout unit with two lines per group can reproduce a rectified unit. This is the book’s claim about two-piece maxout layers. The added derivation behind it is short: , so one line fixed at zero, with the other line equal to the pre-activation, is a rectified unit.

The book also gives a cost and a benefit. Each maxout unit has weight vectors instead of one, so it usually needs more regularization. If the features from filters can be summarized without loss by the maximum over groups, the next layer needs times fewer weights. Because each unit is driven by several filters, maxout units also have some redundancy, which the book says helps them resist catastrophic forgetting, the failure in which a network forgets a task it learned earlier.

Logistic sigmoid and hyperbolic tangent

Before rectified units, most networks used the logistic sigmoid or the hyperbolic tangent as the bend. The sigmoid is

and the tanh is

The two are closely related, since the book gives the identity

Unlike piecewise linear units, these bends saturate over most of their domain. The sigmoid goes towards 1 when is very large and towards 0 when it is very negative. It responds strongly only near zero. Where the bend is flat, its slope is close to zero, so gradient reaches the layers below only weakly. The book says this widespread saturation can make gradient learning very difficult, which is why sigmoidal hidden units are discouraged in feedforward networks. As output units they remain useful, when the cost function undoes the saturation.

Predict first. At , which bend has the steeper slope?

Choose an answer.

The slope of each bend is needed to read the demo. The following two formulas are standard calculus and are added here. The book does not print them in this section.

The book says that when a sigmoidal hidden unit must be used, tanh usually does better. Tanh is zero at zero, while the sigmoid is one half there, so tanh resembles the identity, the function that returns its input unchanged, near zero. This matters for a deep network. The book writes a two-layer tanh network as

and notes that it resembles the linear model when the activations are small. The demo below tests that resemblance with one weight.

This toy sets every weight to one number, , so the network has a single knob. The added check is the relative gap between the two curves. The chain stays within 5% of the line only up to a limit that shrinks as grows.

Other hidden units

The book notes that many differentiable functions work well as hidden units. Many unpublished units perform as well as the popular ones. Testing several variations and finding that they perform alike is common practice. For that reason the book says a new kind of unit is usually published only when it clearly improves results.

A hidden unit can also use no bend at all. The book points out that a linear unit can replace one layer with two. Let map the inputs to numbers and let map those to the outputs. Then the pair computes

The original layer has weights, and the factored pair has fewer when is small. The book adds that the saving costs something: it forces the transformation to be low rank. The two counts below follow the book’s own description, and the demo shows where the pair stops being smaller.

Softmax units are usually outputs, but they can also be hidden units. A softmax represents a distribution over values, so it can act as a switch. The book uses such units in architectures that learn to manipulate memory, which it covers in section 10.12.

The book lists three more reasonably common hidden unit types. The radial basis function unit becomes more active as its input approaches a template, the column of the weights. It saturates to 0 for most inputs, so it can be difficult to optimize.

The softplus is a smooth version of the rectifier. The book says the softplus is generally discouraged. Glorot et al. found better results with the rectifier, and the softplus has a counterintuitive weakness: it is differentiable everywhere and saturates less, yet empirically it does not beat the rectifier.

Hard tanh is shaped similarly to tanh and to the rectifier, but unlike the rectifier it is bounded.

The book gives one concrete example of an unusual bend. It reports a feedforward network with hidden units that reached an error rate below 1% on MNIST. That is the book’s report. This page does not reproduce that result. The demo above does not train a network either. The cosine is shown here only for its shape.

• A hidden unit forms a pre-activation and passes it through a bend. The bend sets the slope that training can use.
• The rectified unit has slope 1 when it is on and 0 when it is off. The book calls it an excellent default.
• A non-zero slope on the negative side of a rectified unit keeps gradient flowing everywhere.
• A maxout unit takes the maximum of its lines, so it can learn a piecewise linear bend with up to pieces.
• Sigmoid and tanh saturate far from zero. Tanh has the larger slope at zero, and it is the better sigmoidal choice.
• A factored layer uses weights against , so it saves weights only when is small.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 6.3 (pages 191–196), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.