Performance Metrics

The error no model can beat

Before you train anything, you need to know what success means. A metric is the number you promise to improve. It guides every later decision, so picking it is the first step of the design process. For most applications, absolute zero error is impossible, so the useful question is how close you can get.

Start with a guess. Two groups of examples overlap, and a rule that sees only one number per example has to pick a side for each. How often would even the best possible rule be wrong? Pick an answer, then check it with the demo.

Pick a guess for the error at separation 2.

Each example is reduced to a single score. Positive examples center on and negative examples on , both with spread 1. The best cut, a threshold on the score, sits at 0. Shaded areas are the examples the cut gets wrong. The demo is a toy with equal numbers of each class, and the formula below is derived here for that case.

(This derivation is added here and is not printed in the book.) The result is the Bayes error, the lowest error that any rule can reach. The book defines it as the minimum error rate you can hope for, even with infinite training data and the true probability distribution known. It is positive for two reasons the book names: the input features may not contain the full information needed to get the output right, and the process may be random in its own right. Here the two bell curves overlap, so no cut separates them perfectly. A forecaster with perfect knowledge of the atmosphere still cannot be certain about tomorrow, for the same reason.

Two more limits come from the data. The book notes that training data is finite, and that collecting more costs time, money, or in some medical settings human suffering. The book also points out that a benchmark fixes the training set, so no more data may be collected. Your target error comes from one of two sources: what the published benchmarks reach, or what the application needs to be safe, cheap, or attractive to users. The target then guides every design decision.

The number you report is often different from the number you train on. Training uses a cost function that the optimizer can follow. The metric is what the product needs. The book uses accuracy, or equivalently the error rate, as the usual example. The next sections show when that choice is wrong.

Hold on to the Bayes error as the floor for the error rate. No rule can push the error rate below it, however much data it has. The gap between your model and this floor is what better data and better methods can close.

Error rate and the cost of each mistake

Start with the simplest metric. The error rate is the share of examples the system gets wrong, and accuracy is the share it gets right. The two always add up to 1.

(These two lines are added here as plain definitions. The book uses the same two quantities with the same meaning.) Accuracy treats every mistake the same. A spam filter shows why that is not always right. It can make two kinds of mistake: a legitimate email goes to the spam folder, or a spam email reaches the inbox. The first mistake is usually far worse, because a missed invoice or a job offer is a loss, while a spam message in the inbox is only a nuisance. The book suggests measuring total cost, where blocking legitimate mail costs more than letting spam through.

Here counts the legitimate emails blocked (false positives), counts the spam let through (false negatives), and the two prices and say how bad each mistake is. Choosing a cut for the score is then a question of which total is smallest.

This is a toy filter. Legitimate emails get a score from a standard normal distribution, spam gets one centered at 2, and half of the mail is spam. The price of letting one spam through is 1. Raise the price of blocking a legitimate email and watch the cheapest threshold move.

The demo plots the expected cost per email, with half of the mail spam:

Here is the share of legitimate emails blocked, is the share of spam let through, and is the price of a blocked legitimate email. Spam costs 1 each time it gets through, and each rate is weighted by the half of the mail that is of its kind.

The accuracy trap for rare events

Suppose a disease affects one person in a million. A detector that always says “healthy” is wrong only for the single sick person. Its accuracy is , or 99.9999%. The book makes the same point: accuracy is a poor way to describe a detector for a rare event, because a useless detector can score almost perfectly. The share of people who have the condition is the prevalence, . Here it is 0.000001.

The book proposes two replacements. Precision is the fraction of the detections that were correct. Recall is the fraction of the true cases that were detected. In the same notation as the counts above:

(These two lines are added here, written from the book’s words.) Here counts the sick people found, the sick people missed, and the healthy people flagged. Both quantities lie between 0 and 1.

The book states two extremes for the disease example. A detector that reports nobody has perfect precision and zero recall. A detector that reports everybody has perfect recall, and its precision equals the share of people who are sick. This derivation is added here: the always-absent detector reports nothing, so its precision has a zero denominator. The book calls its precision perfect. Strictly, 0 divided by 0 has no value, so the demo marks it as not defined, and calling it perfect is a convention. The point stands either way, because its recall is zero.

Click through the three detectors at the default one in a million. The always-healthy detector has the highest accuracy and no recall at all. Then set the prevalence to 1 in 1,000 and move the test’s sensitivity down to 0.5. The real test now has a lower accuracy than the always-healthy detector. Recall shows the test finds half the cases, something accuracy hides completely.

Accuracy depends on the prevalence strongly, and precision depends on it too, while recall does not. That is why the same detector can look excellent on one population and poor on another.

Precision, recall and the PR curve

Many detectors do not answer yes or no directly. The book describes a network that outputs a score, , which estimates the probability that the example is positive. You report a positive whenever the score is above a threshold . Lowering finds more true cases, and raising it reports fewer, more careful positives. The threshold therefore trades precision for recall.

Plot precision against recall as the threshold sweeps from high to low, and you get the precision-recall curve, or PR curve. The book uses precision on the vertical axis and recall on the horizontal axis. The demo builds the curve from two score distributions, so you can see the trade-off directly.

The negative examples have scores from a standard normal distribution. The positive examples have the same spread, centered at . The share of positives reported is recall, and the probability of reporting a negative is the false positive rate . Both depend only on the threshold and the separation:

(These two lines are added here. They are the normal tail probabilities of the demo, written with , the standard normal distribution function.) Precision then follows from the prevalence , because a report is correct with probability proportional to the positive share:

(This formula is added here and is not printed in the book.) A detector that ranks positives well shows a curve that stays near the top right corner. A detector that ranks at random gives a flat line at the prevalence.

One number: the F-score and the area under the curve

A curve is informative, but a single number is easier to compare across models. The book offers two. The first is the F-score, which combines precision and recall into one value between 0 and 1:

In words, is twice the product of precision and recall, divided by their sum. It is the harmonic mean of the two numbers, an average that is pulled towards the smaller one. The second option is to report the total area under the PR curve, which summarizes the whole trade-off without fixing a threshold.

Predict before you slide. Imagine a precision of 0.9 and a recall of 0.1, and guess the F-score with the buttons below. The demo then sets those two values and shows the answer.

Make a guess for the F-score when precision is 0.9 and recall is 0.1.

Now the arithmetic. The harmonic form below shows why the smaller number dominates. One precision of 0.9 combined with a recall of 0.1 gives an F-score near 0.18, not the plain average of 0.5. (The harmonic form is added here and is not printed in the book.) An F-score is high only when both numbers are high.

The area under the PR curve has its own baseline. A ranking with no information has the same precision at every threshold, equal to the prevalence . The area under that flat curve is therefore . (This is added here and is not printed in the book.) A useful detector has an area well above the prevalence, and the value of one is the best possible.

Refusing to answer: coverage

Some systems may decline to decide. The book’s running example is the address transcription system used by Street View. The system reads the house number from a photograph, and the map adds the address only when the reading is right. If the system is not confident, a human takes over. The system is useful only if it removes most of the work from the human operators.

The metric for that situation is coverage, : the fraction of examples the system answers. Refusing every example gives 100% accuracy and 0% coverage, so accuracy and coverage must be read together. The book states the goal of the Street View project as human-level transcription accuracy, 98%, while keeping 95% coverage.

(This line is added here as a plain definition.) The demo uses a toy table, invented for this page and not taken from the Street View data. The system ranks its answers from most to least confident, and each bar is one twentieth of the photos. Move the slider to choose how many bars it answers.

Each step to the right adds more photos and usually lowers the accuracy of the answers taken together. The trade is the whole story. Answering fewer photos usually raises accuracy, and answering more covers more of the work. Choose the operating point by the cost of a wrong address and the cost of a human reading.

Choosing the metric before tuning

The book’s advice is to decide the performance metric before you improve the system. Many other metrics exist. A recommendation system might use click-through rates. A product might use user satisfaction surveys. Without a defined goal, you cannot tell whether a change is progress. Try the game below. Five situations each need the right metric. Read the situation, then pick one answer. You get one try and a second try, after which the answer is shown.

A metric answers one question about the system. Accuracy answers how often it is right, precision and recall answer how trustworthy its positives are and how many positives it finds, the area under the curve answers how well it ranks, coverage answers how much it is willing to decide, and total cost answers what the mistakes are worth. Use the one that matches the question the product must answer.

• The Bayes error is the lowest error any rule can reach. Overlap in the inputs, or randomness in the outputs, makes it positive even with infinite data.
• Accuracy can mislead when positives are rare. A detector that always says no can score 99.9999% and find nothing.
• Precision is the share of reports that are right. Recall is the share of real cases found. A threshold on the score trades one for the other.
• The F-score (equation 11.1) is one number that needs both precision and recall to be high. The area under the PR curve summarizes the trade-off over all thresholds.
• Coverage is the share of examples a system answers. Accuracy and coverage go together, and the book’s Street View goal sets both.
• Decide the metric and its target before changing the model, so that each change can be judged against them.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 11.1 (pages 422–424), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.