Useful Properties of Common Functions

The logistic sigmoid

A network ends with a raw score . It can be −40 or 12, and it is not yet a probability. This section turns any score into a probability between 0 and 1.

The book’s tool for this is the logistic sigmoid, . Equation 3.30 defines it:

Think of a dimmer switch. Turned far in either direction, the lamp is already fully off or fully on, so the brightness barely changes. Only near the middle does a small turn change the light. The score is the knob, and the probability is the brightness.

The book uses the sigmoid to produce the Bernoulli parameter (Section 3.9). Its range is the open interval (0, 1), exactly the range must lie in.

Read the formula from the inside. When is very large and positive, is nearly 0, so the value is nearly 1. When is very large and negative, is huge, so the value is nearly 0. At the value is exactly one half.

Score 0 gives a fair coin. Positive scores favor outcome 1, negative scores favor outcome 0, and the two bars always add up to 1. The readout also gives the odds, the ratio of to . The odds equal , so the score is the logarithm of the odds.

A second way to write the same function

The book also writes the sigmoid as Equation 3.33:

Since is 1, the denominator is . Read the form with as a two-way contest. The score competes with a reference score of 0, and each takes a share in proportion to its exponential.

Saturation

Drag the score away from 0 and watch the tangent line flatten. The book calls this saturation: the sigmoid becomes very flat and insensitive to small changes in its input when the score is very positive or very negative.

The slope is the derivative. The book gives it in Equation 3.34, in a form that is easy to remember:

The slope is the value times its complement, . It is largest when , which is at , where the slope equals 1/4. Far from 0 it shrinks to almost nothing.

A parameter changes in proportion to the slope of the function it passes through during training. A unit on a flat part passes back almost no signal, and a chain of units multiplies their slopes. This paragraph is added here as an illustration, and the book does not state it. At score 0 each unit scales the signal by at most 1/4, so ten units scale it by about 10⁻⁶.

Why flatness matters when sigmoids are stacked

Saturation is not a defect of the sigmoid as a probability model. It becomes a problem only when the sigmoid is a hidden unit in a deep stack, where a small slope at every layer shrinks the signal layer after layer. Later chapters return to this trade-off.

The softplus function

Pick a sharpness and watch a smooth curve close in on a sharp corner. The second function of the section is the softplus, Equation 3.31, which the book attributes to Dugas and colleagues (2001):

Its range is (0, ∞). That makes it useful for a scale. The book notes that the softplus can produce the precision or the standard deviation of a normal distribution, because a network can output any real score and the softplus turns it into a positive value.

The name explains the shape. Softplus is a softened version of the positive part, Equation 3.32:

The positive part passes positive numbers unchanged and sets negative numbers to 0, so it has a sharp corner at 0. The softplus follows it far from 0 and rounds the corner. At the softplus equals log 2, about 0.693.

The largest gap sits at the corner, where . There the gap equals log 2 divided by , so a sharper curve has a smaller gap.

A numerical warning (added here, not in the book)

The formula fails for large on a computer. The exponential of 710 is larger than the largest number double precision can store, so the direct formula loses its answer. A stable form avoids exponentiating a positive number: .

Identities for sigmoid and softplus

Pick an identity, move the derivation step, and read the proof one line at a time. The book says the following properties are useful enough that you may wish to memorize them. The book states them without proof, so the proofs here are added and are easier to remember once derived.

Equation 3.34, the slope of the sigmoid, was shown in Section 2. Equation 3.35 says the sigmoid of a negated score is the complement. Equation 3.36 says the log of a sigmoid is a negative softplus. Equation 3.37 says the slope of the softplus is the sigmoid.

Inverse functions

Start at a score, push it through the sigmoid, then push the result back. Do you return to where you started? Both functions can be undone. The inverse of the sigmoid takes a probability and returns the score that produced it, Equation 3.38. The inverse of the softplus takes a positive number and returns the score, Equation 3.39.

The letter in these two equations is the argument of the inverse. It is a probability in (0, 1) for the first and a positive number for the second. The book uses the letter for both, and here it is not a score.

The book notes that is called the logit in statistics, a term that is more rarely used in machine learning. The logit is the logarithm of the odds that Section 1 introduced.

The two curves in each tab are mirror images across the dotted diagonal. Apply the forward function, then the inverse, and you return to the score, up to arithmetic precision. Equation 3.38 needs , so the logit tab cannot take a probability of 0 or 1.

Integral and the positive part

Shade the area under the sigmoid from far left up to a point, then compare it with the softplus at the same point.

The softplus is the accumulated area under the sigmoid. Equation 3.40 writes that as an integral, and the book states it without proof:

Equation 3.37 gives the slope of the softplus as the sigmoid. Integrate that slope from far left up to , and the result is the softplus at , because the softplus is 0 at the far left. That is why the area equals the softplus.

The book uses Equation 3.41 to justify the name. The positive part is , and the negative part is = max(0, −). A number is recovered from its two parts, so .

The softplus has a smooth counterpart for the negative part, and the same relation holds for it:

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 3.10 (pages 67–70), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.