Default Baseline Models

The first system to build

Before you tune anything, build one working system from the input to the output. The book calls this the baseline: a reasonable model that runs on real data and gives you a number to beat. Think of it as a first draft of a recipe. A rough version you can taste teaches you more than a perfect plan you cannot test. The book adds that research moves quickly, so better default choices will appear, and the defaults below are a starting point.

The book gives a rule for the first draft. The family of model depends on the shape of the data. Each kind of input has a standard starting point. Match the four inputs below to the family the book recommends, then read the reason it gives.

If your input is a fixed-size vector, use a feedforward net with fully connected layers. If the input has a known topological structure, such as an image, use a convolutional net. If the input or output is a sequence, use a gated recurrent net, an LSTM or a GRU. If your problem may be solved by choosing a few linear weights correctly, start with a simple statistical model such as logistic regression. The book adds that an “AI-complete” problem, such as object recognition, speech recognition or machine translation, is likely to go well with an appropriate deep learning model from the start.

(The four inputs and their reasons are the book’s recommendations, rewritten here. The rows are one example each and do not cover every case.)

Piecewise linear units to start with

Inside each layer, a unit turns its input into an output with a nonlinear function. The book recommends starting with a piecewise linear unit, a unit that is a straight line on each side of one kink. The rectified linear unit, or ReLU, is the standard example. Its output is zero for negative inputs and equals the input for positive ones.

The book also names the generalizations to try: leaky ReLUs, PReLUs and maxout units. A leaky ReLU keeps a small slope for negative inputs. Here is that unit as one equation.

(The description of the PReLU and the maxout unit is added here. In a PReLU the slope is learned, and a maxout unit takes the largest of several linear pieces. The book names both without these details.) Move the input and the slope to see why the leak matters.

The gradient of the unit is the slope of this line at the input. For a ReLU it is 1 on the right and 0 on the left. A unit with gradient 0 gets no learning signal from that input, so the weights below it cannot move. The leaky version passes a small gradient on the left, so the same unit keeps learning there.

Learning-rate schedules

The book recommends stochastic gradient descent with momentum, and a learning rate that decays. An epoch is one full pass over the training set, and validation error is the error on held-out examples that training does not use. The learning rate is the step size of each update. The starting value is . Three decay schemes are named in the book: decay linearly until a fixed minimum rate is reached, decay exponentially, or divide the rate by a factor of 2 to 10 each time validation error plateaus. The book also names Adam as another reasonable choice.

Predict before you open the demo. The exponential schedule starts at and multiplies by = 0.97 each epoch. Which share of the starting rate is left at epoch 100?

Pick a guess.

(The stepwise schedule here drops the rate after a fixed number of epochs. The book describes drops each time validation error plateaus. The fixed interval is a stand-in added here.) Each scheme is written below with the same starting value .

In the exponential case the rate at epoch is times to the power . A of 1 keeps the rate constant, and a smaller value makes it fall faster. These three formulas are added here to make the curves exact. They are not printed in the book.

Early stopping

The book says early stopping should be used almost universally. Training error keeps falling. Validation error falls for a while, then rises as the model starts to fit the training set’s noise. The rule is to keep the model from the best epoch seen so far, and to stop once a number of epochs called the patience has passed with no new best.

Predict before you open the demo. The patience is 3, and the other settings are at their defaults. Which epoch does the model keep?

Pick a guess.

The curves are a toy. Training error falls with time. Validation error falls, then rises at the rate set by the slider. A small wiggle is added to the validation curve here to mimic noisy measurements. This wiggle is added here and is not in the book. The rule being taught is the same whatever the curve.

(This stopping test is written here from the book’s words and is not printed in it.) Here is the current epoch and is the best epoch so far. The kept model is the one from , not the one at the stop. Training runs on for more epochs before it stops, so the stop epoch is always later than the kept one.

Regularization and batch normalization

The book adds two defaults for the first baseline. The first is regularization, any change that makes a model generalize better, such as keeping the weights small. Early stopping from Section 4 is one form. The book says that unless the training set contains tens of millions of examples or more, you should include some mild forms of regularization from the start.

Dropout is a regularizer that randomly switches off units during each training step. The book calls it an excellent regularizer that is easy to implement and compatible with many models and training algorithms. (The short descriptions of regularization and dropout are added here. The book names both without a description in this section.)

The second default is batch normalization. The book says it can have a dramatic effect on optimization performance, especially for convolutional networks and networks with sigmoidal nonlinearities. It is reasonable to leave it out of the very first baseline, but introduce it quickly if optimization appears to be problematic. It also sometimes reduces generalization error, which can allow dropout to be omitted, because the statistics used to normalize each variable are estimated from noisy minibatches.

The idea is to rescale each variable with the mean and spread of the current minibatch, so that the next layer sees values in a stable range. The demo shows the operation on one variable.

For a minibatch of values , the mean and the spread are measured on that batch, and each value is rescaled to . The constant is tiny and only stops the division by zero. These equations are added here and are not printed in this section.

Batch normalization can change optimization a lot, so it is worth trying when the baseline does not train well. It is not a reflex for every model.

Reuse, copying and unsupervised starts

Before you design anything new, check whether the task has a close relative that is already solved. The book says that if your task is similar to one studied extensively, you will probably do well by copying the model that performs best there. You may even copy a trained model. A common case is the features of a convolutional net trained on ImageNet, reused for other vision tasks.

Whether to start with unsupervised learning depends on the domain. The book says natural language processing benefits a great deal from unsupervised word embeddings. In computer vision, current unsupervised methods bring no benefit, except in the semi-supervised setting when labeled examples are very few. If the task itself is unsupervised, use unsupervised learning from the start. You can add it later if the baseline overfits.

The advice above depends only on the four answers. The first answer is about a similar task, the second and third about the domain and the task, and the fourth about how many labels you have. The rules are the book’s advice rewritten as a checklist (this checklist is added here and is not in the book). Toggle the answers and watch which lines change.

Summary

• Build an end-to-end baseline first. Choose the family by the shape of the input: fully connected layers for vectors, convolution for images, gated recurrent nets for sequences.
• Use piecewise linear units such as ReLU, and a leaky version if units stop learning.
• Use stochastic gradient descent with momentum and a decaying learning rate, or Adam, and add early stopping from the start.
• Include mild regularization and early stopping from the start, and introduce batch normalization when optimization looks problematic.
• Copy the best known model when a similar task has one, and include unsupervised learning when the domain or the task calls for it.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 11.2 (pages 425–426), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.