Selecting Hyperparameters

Hyperparameters and capacity

A guitar has tuning pegs. You turn each one before you play, and the sound depends on those settings. Hyperparameters play the same role for a learning algorithm. They are the settings you choose before training, such as the learning rate, the number of hidden units or the weight decay. Training then fits the parameters , the weights and biases, to the data.

The goal is the lowest generalization error that fits inside a runtime and memory budget. The main lever is capacity, the range of functions the model can represent and fit. Too little capacity underfits, and too much overfits. Effective capacity is limited by three things: the representational capacity of the architecture, the ability of the learning algorithm to minimize the cost, and the regularization built into the cost and the training procedure. A model with many hidden units can still learn a simple function if training cannot find the complicated one. The first demo lets you turn one capacity dial and watch both errors.

Here is the capacity dial, a toy built for this page. The model is a polynomial of degree fitted to 12 noisy points taken from the curve . Before you touch the slider, predict which degree gives the lowest test error.

The U-shape is the point of this demo. At low degree both errors are high, because the model is too simple to follow the curve. At high degree the training error keeps falling while the test error climbs, because the model starts to follow the noise in the 12 points. The book describes the same U-shape for a hyperparameter in its Figure 5.3. The position of the minimum, here degree 5, belongs to this toy data set and is not a rule. Between degrees 3 and 7 the test error is almost flat, so a different noise draw would move the minimum.

The test error is what you care about. The book splits it into the training error plus the gap between the two:

Notice what the equation says. A model can lower its training error and still raise the test error, if the gap grows faster.

The learning rate curve

The book calls the learning rate perhaps the most important hyperparameter, and advises tuning it first if time is short. Its effect is not a simple slope. Too small a rate makes training slow, and in non-convex problems it can leave training stuck at a high error. Too large a rate makes gradient descent overshoot, so the training error can rise. Between the two lies a good range. Figure 11.1 of the book shows training error against the learning rate on a logarithmic scale. The error falls as the rate grows, then rises sharply above an optimal value. The figure is drawn for a fixed training time, because a smaller rate may only slow training by a factor proportional to the reduction.

Rebuild that picture with the simplest case that has an exact answer: a quadratic loss with curvature . Gradient descent with rate takes a step that shrinks the parameter by the factor each time. Set the learning rate and the number of steps , then watch the error after those steps.

Try two things. Slide the rate past 2/λ and watch the error stop shrinking and then grow. Between 1/λ and 2/λ the error still falls, but each step overshoots the minimum. Then set the steps to 1, slide the rate down to 0.02, and see how little one step achieves. The curve in the demo is exact for the quadratic loss, not a measured network curve.

For this quadratic loss the derivation is short, and it explains where the threshold comes from. This derivation is added here and is not printed in the book. The loss is , so each step multiplies the parameter by . After steps the error has been multiplied by:

The error shrinks while . The best rate is , where one step lands exactly on the minimum. When , the factor has magnitude at least one, so the error no longer shrinks. In this idealized case the training error stops shrinking once the rate reaches twice its optimal value, and grows beyond it. The threshold follows directly from the factor above:

Set the rate to its smallest value, 0.02, and notice how little one step achieves. A rate that is too small is a real risk in non-convex problems, where the book notes that training may stop at a high error. The quadratic here is convex, so it only learns slowly.

Which knob does what

Most hyperparameters can be set by asking one question: does this knob increase or decrease the capacity of the model? The book gives six examples in its Table 11.1, which lists the direction that raises capacity and the reason for it. Sort them yourself before you check the table.

The learning rate is the exception. Its capacity is highest at a tuned value, because a rate that is too large or too small fails to optimize. The other five move in one direction. Dropout follows the same rule: a lower dropout rate lets more units fit the training set, so capacity rises.

Some knobs can only remove capacity. Weight decay cannot go below zero, so if the model underfits with zero weight decay, no other setting of that knob reaches the overfitting region. The book calls these hyperparameters that can only subtract capacity. Discrete and binary hyperparameters have a further limit: the search visits only a few points on the U.

Reading the two errors

Tuning every other hyperparameter means reading two numbers. If training error is above your target, the model lacks capacity or the optimizer fails. If training error meets the target but test error does not, the gap is the problem. The book states these two actions. The sorting logic follows its wording. The demo uses your target as the threshold for both errors.

The book adds that once the training error is low, you can always reduce the generalization error by collecting more data, and that a very large regularized model is the usual best choice. The diagnosis is therefore a loop. Move one knob, read both errors again, and stop when both meet the target.

Automatic search and grid search

Manual tuning needs understanding and experience. Logistic regression and SVMs need only one or two tuned hyperparameters, but neural networks often benefit from tuning forty or more. Automatic methods treat the choice as an optimization. The decision variables are the hyperparameters, and the cost is the validation error, sometimes with a budget for training time or memory. In symbols, the search looks for the best joint configuration. This line is a summary of the book’s wording and is not printed in it:

The automatic method has its own settings, such as the range to explore for each hyperparameter. These secondary settings are usually easier to choose, because a single set of them often works across many tasks.

Grid search

When there are three or fewer hyperparameters, the usual practice is a grid. You choose a short list of values for each one, and the search trains a model for every combination. With hyperparameters and values each, the number of trials is

The cost grows exponentially with the number of hyperparameters. The book notes that the trials can run in parallel without much communication, but even so the count soon outgrows the budget. The lists are usually spaced on a logarithmic scale, such as for a learning rate. If the best value sits at the edge of a list, shift the range and search again. If it sits in the middle, zoom in.

The lesson is the multiplication. Each new hyperparameter multiplies the trials by , and the bars above grow in steps of the same height. Grid search wastes its budget on all but a few of these combinations, as the next section shows.

Random search on a log scale

Random search is as simple to program as a grid, more convenient to use, and it converges much faster to good values. The book cites Bergstra and Bengio (2012), who found that random search reduces validation error much faster than grid search as the number of trials grows. The method first defines a distribution for each hyperparameter, then draws joint configurations from those distributions. Unlike a grid, the values are never rounded to a fixed list, and repeated runs can refine the range, as with a grid.

For a positive hyperparameter whose sensible range spans several powers of ten, the book samples the exponent uniformly, then raises ten to that exponent:

The book writes the interval as u(−1, −5). Read as a sampling interval, it runs from −5 to −1, which gives learning rates from 10⁻⁵ to 10⁻¹, the same range as the grid example. This reading is added here and is not printed in the book. The book samples the number of hidden units the same way, uniformly between log 50 and log 2000.

On the log scale the draws spread evenly over the decades, so each decade gets a quarter of the draws. On a linear scale the same range is dominated by its top decade. A learner who draws linearly tries almost nothing near 10⁻⁴. The bottom row shows this. Only about one draw in ten lands below 10⁻², while the log-uniform draws put about three in four there.

Why random search wastes fewer trials

Figure 11.2 of the book compares the two searches when only one hyperparameter matters. The horizontal axis carries the important hyperparameter. A grid reuses the same few values of it in every row, so most of its trials repeat the same information. Random search gives every trial a new value of the important hyperparameter. Rebuild the picture here with a toy score, chosen for this page and not taken from the book. The score depends only on , with a peak at 0.7. The second hyperparameter does not matter.

The grid with values per side spends trials but tests only distinct values of . Random search with the same number of trials tests as many distinct values as it has trials. When the other hyperparameters have little effect, this is the difference that matters. The book attributes it to the absence of wasted runs.

Model-based search and the takeaways

The search can also be cast as optimization over the hyperparameters. If the gradient of the validation error with respect to the hyperparameters were available, you could follow it. In most practical settings it is not, because it is too expensive or because some hyperparameters are discrete. So model-based methods build a model of the validation error and propose new guesses by optimizing inside that model.

Most of these methods use a Bayesian regression model. It gives an expected validation error and an uncertainty at each point. The search then trades exploitation, trying points the model expects to be good, against exploration, trying points where the model is unsure and a large improvement might hide. The book names three systems, Spearmint, TPE and SMAC. It also states that the book cannot yet unambiguously recommend Bayesian optimization as an established tool for better deep learning results. It sometimes matches human experts and sometimes fails badly.

One drawback is structural. Most of these methods wait for each training run to finish before they learn anything, while a human can often see early that a setting is hopeless. The book mentions a method that can freeze and resume runs, and notes that it is an early version.

A small model of the validation error

The demo below uses a toy Gaussian process, built for this page. It is not one of the named systems. The horizontal axis is , and the curve is a toy validation error that is lowest near −2.5. Each click adds a trial. The violet line is the model’s estimate , and the band shows the estimate plus or minus two times its uncertainty . The suggested point is the lowest value of , where sets how much uncertainty counts.

Click the plot to add your own trials, or press the suggestion button to let the model choose. With a small the model keeps returning to the region it already rates best, and it can suggest a point it has already tried. With a large it goes looking in the regions it knows least about. Both are useful. The art lies in choosing a value of that suits the problem.

Takeaways

• Hyperparameters are chosen before training. They set the capacity of the model and how the parameters are fitted.
• The U-shape of generalization error against capacity is the central picture. Read training and test error together.
• The learning rate matters most. Its curve is best near one over the curvature, and it fails above twice that value in the quadratic case.
• Grid search grows as n to the power m. Random search tests a new value of every important hyperparameter on each trial.
• Sample scale-like hyperparameters on a logarithmic scale, as in equations 11.2 and 11.3.
• Model-based search is promising but not yet a reliable default. Its estimates need care, and it waits for runs to finish.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 11.4 (pages 427–435), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.