Section 1
Finite numbers for infinitely many reals
, , scores, index, count
the main object
, shifted scores, largest score
, , , small constants
loss and errors
How to use this pageClick any colored symbol. Inside an equation its meaning appears under the equation, and in running text a small note opens. Answer each prediction before you move the controls, then read the sentence under the demo.
Predict first. You add 10−9 to 1 in float32, then subtract 1. What comes back?
Mathematics uses every real number. A computer stores a number in a fixed count of bits, so it has only a finite list of values to choose from. Most real numbers must be replaced by the nearest stored value, and that replacement is rounding error.
Picture a ruler whose marks crowd together near zero and spread apart far from it. Every number sits near some mark, and the spacing between marks is the precision at that size. One rounding is harmless. Across millions of operations the errors add up, which is why they need attention.
Drag the small number off the ruler
The chart plots the share of that comes back after computing . Each curve stays at 1 while is large enough to move the sum off 1, then drops to 0. Below that point the small number is absorbed. The threshold is set by the gap between 1 and the next stored number, and each format has its own gap.
Challenge +8 XPDrag so that float16 loses the small number completely while float32 still keeps it. The readout shows both.
| Format | Bits | Largest | Smallest normal | Smallest subnormal | Gap above 1 |
|---|
| float16 | 16 | 6.55e4 | 6.10e-5 | 5.96e-8 | 9.8e-4 |
| float32 | 32 | 3.40e38 | 1.18e-38 | 1.40e-45 | 1.2e-7 |
| float64 | 64 | 1.80e308 | 2.23e-308 | 4.94e-324 | 2.2e-16 |
The table lists the common IEEE 754 formats. The book describes the principle but does not print these values. The last column is the gap above 1, which is the machine epsilon .
Going deeper: why the threshold is half the gap (added here, not printed in the book)
Stored numbers near 1 are apart. The sum rounds back to 1 whenever is less than half that spacing, which gives the rule below. Exactly half is a tie, and the rounding rule for ties decides that case. A subnormal number is a very small value that gives up some precision to sit closer to zero.
Click any coloured symbol to see what it means.
Underflow and overflow
Predict first. In float32, what does give when ?
Underflow happens when a number near zero is rounded to zero. Overflow happens when the magnitude is larger than the largest stored number, and the result becomes . The book calls both damaging, and the reason is what they do to the next operation.
The exponential causes both problems most easily. Its range of safe inputs is narrow, and the demo below finds the edges for each format.
Where exp survives
usablesubnormal, precision lostzero or infinite
Each band is one format. The teal part is the set of inputs for which exp gives a usable number. In float32, already overflows at , and it underflows at . A logit of 100 in a network is not unusual (this remark is added here), so the danger is real. Even float64 overflows beyond about 709.
Challenge +8 XPDrag to a value where float16 overflows but float64 stays normal. The status column shows both.
- Many functions behave differently at zero than at a tiny positive number. Dividing by zero raises an error in some software and returns a placeholder in others. The logarithm of zero is usually treated as minus infinity.
- Arithmetic on infinity often has no value, such as infinity minus infinity, or infinity divided by infinity. The placeholder for such a result is not-a-number.
- The placeholder spreads. Any later operation on it returns it again, so one overflow can corrupt every later result that uses it (this consequence is added here).
Predict first. The next demo applies five rules to , which is large enough that overflows in float32. What does return?
One bad value, five wrong answers
| Expression | Should be | Computed |
|---|
When is very negative, exp underflows and the reciprocal divides by zero. When is very large, exp overflows and the other rules fail. Each rule is correct in exact arithmetic, so the finite range of the format is the only cause.
Challenge +8 XPPush until all five rows are wrong at the same time. Try float32 first.
Softmax with equal inputs
The softmax function turns real scores , called logits, into positive numbers that add up to 1. Those numbers are read as the probabilities of a multinoulli distribution, which has exactly one outcome out of . The book gives the definition as Equation 4.1:
Click any coloured symbol to see what it means.
Read it from the inside out. is the score of outcome . The exponential makes it positive, and a larger score gets a larger value. The denominator adds the exponentials of all the scores, so the outputs add up to 1.
Predict first. Here and every score equals . The direct formula is computed in float32. What comes out?
Take the simplest input: every score equals the same constant . Nothing favors any outcome, so each probability should be .
Click any coloured symbol to see what it means.
(This line is the symmetry argument written out, and it is added here.) The factor cancels, so on paper does not matter. On a computer it matters a great deal.
Softmax of equal scores, computed directly
direct formulacorrect valueshort red ticks: not-a-number
The orange line is the first output of the direct formula, and the teal line is the correct answer. The formula works in a window around zero. Where is large and positive, the exponential overflows and the result has no value. Where is very negative, every term underflows to zero and the result again has no value. The red ticks mark those inputs. The window is much narrower in float16.
Challenge +8 XPDrag until the direct formula returns not-a-number. Start from zero and move far in either direction.
The shift that stabilizes softmax
Predict first. Take the offset in float32 and set the shift . The four probabilities are computed directly. What do you get?
The fix is to subtract the largest score before taking exponentials. Define the shifted scores
Click any coloured symbol to see what it means.
and evaluate in place of . The shifted vector has the same shape. Its largest entry is exactly 0, and every other entry is at most 0, negative unless it ties the maximum.
Why the answer does not change
Softmax is unchanged when any number is subtracted from every score. The algebra uses one exponential rule only.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
Click any coloured symbol to see what it means.
The factor cancels between numerator and denominator. The output depends only on differences between scores. The book states this invariance, and the derivation is added here.
Going deeper: why the maximum is the right shift
Any shift keeps the answer, so the choice is made for numerical reasons. With , two things follow. The largest exponent is 0, so is 1 for the entry that equals the maximum, and nothing overflows. At least one denominator term equals 1, so the denominator is at least 1 and cannot underflow to zero.
Choose the shift yourself
exactcomputed
With equal to 0 and a large offset, the computation breaks as in Section 3. A shift far above the maximum is also harmful, because every exponent becomes a large negative number and all the terms underflow. The maximum is the largest shift that keeps every exponent at or below zero.
Challenge +8 XPWith at least 100 in float32, set by hand with the slider or the keyboard (the buttons do not count) so that the computed probabilities match the exact ones to four decimals.
Log softmax
Training usually needs the logarithm of a softmax output, because a classifier’s loss is commonly the negative log-probability of the correct class (this is added here). The book warns that the shift does not finish the job. Underflow in the numerator can still make the whole fraction zero.
If you compute softmax first and then take its logarithm, a rare class can come out as minus infinity, even though its true log-probability is an ordinary number such as minus 120. The fix is a separate function that takes the logarithm before any division is evaluated.
Predict first. Two outcomes have scores with , computed in float32. The rare probability is about , with . Computing softmax first and then its logarithm returns what?
Click any coloured symbol to see what it means.
The logarithm of a quotient is a difference of logarithms, and the logarithm of an exponential is its exponent. Applying the shift gives the stable form:
Click any coloured symbol to see what it means.
The book describes this method in words, and the displayed formulas are written out here. The numerator is with no exponential, so nothing can underflow there. The sum inside the logarithm is at least 1, so the logarithm never sees zero.
Log-probability of a rare outcome
exactsoftmax, then loglog softmax formula
The orange curve follows the exact line until the rare probability underflows. From that gap on it drops to minus infinity, and red ticks mark the gaps where this happens. The violet curve uses the stable formula and stays on the exact line across the whole range.
Challenge +8 XPDrag past the point where softmax, then log gives minus infinity in float32, while the violet formula still gives a finite value.
Letting libraries do the work
The book does not detail every numerical safeguard. It asks developers of low-level libraries to keep numerical issues in mind, and it tells most readers to rely on libraries that already provide stable implementations. Theano is named as software that finds and rewrites many unstable expressions automatically.
Current frameworks offer combined functions, such as a softmax with cross-entropy loss, built on the stable log softmax (added here). The practical rule is to pass raw logits to such a loss, and not to compose softmax and logarithm yourself.
Predict first. The true class has logit minus 120, and the other two logits are 10 and 5. The loss is computed as softmax, then logarithm, in float32. What comes out?
Cross-entropy with a confident mistake
two-step pipelinestable loss
The stable loss grows in a straight line as the model becomes more wrong, which is what a loss for a confident mistake should do. Its gradient with respect to the logits is the vector of probabilities minus the one-hot target, and that gradient stays finite (this standard result is added here).
Challenge +8 XPLower the logit of the true class until the two-step pipeline reports an infinite loss while the stable loss stays finite. Class 1 is selected at the start.
Going deeper: the gradient with respect to the logits (added here)
Differentiating the stable loss gives the rule below. Each entry is a difference of two numbers between 0 and 1, so the gradient cannot overflow.
Click any coloured symbol to see what it means.
The formula is not printed in the book. It is a standard result for softmax with cross-entropy.
What to take away
- Computers hold finitely many numbers. Underflow rounds tiny values to zero, and overflow turns huge ones into infinity.
- Softmax of equal inputs is on paper, but the direct formula fails when the common score is very large or very negative.
- Subtract the maximum: . The softmax output is unchanged, and the largest exponent becomes 0.
- For log softmax, use and never take the logarithm of a softmax.
- In practice, use the library’s stable fused functions on raw logits.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 4.1 (pages 80–81), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.