Semi-Supervised Learning

Labeled and unlabeled examples

A teacher has marked 20 essays and has 980 that are still unmarked. The unmarked essays are not wasted. They show which essays look alike and where the natural groups lie. Semi-supervised learning uses both piles. It is semi-supervised because only some of the examples carry the answer.

The input is , the thing the model sees. The label is , the answer we want. A labeled example carries both. An unlabeled example carries only . Three probability terms describe the data. is how often each input occurs, and is how often each input and label occur together. The task is to learn , the chance of each label once the input is known.

(The book introduces these three distributions in prose. The display is added here so that the symbols can be clicked.)

Now the demo. There are two clusters of points. Only the points you label carry an answer, and the classes are the two clusters. One label per cluster gives the model very little to work with, so the question is how much the unlabeled points can add.

The label-only rule gives each test point the label of the nearest labeled point. With two labels its boundary is the line halfway between them. It ignores the unlabeled points completely. The representation rule uses the unlabeled points to find the direction of largest spread, and then uses the labels only to place a threshold along that direction. Both rules are scored on a separate set of 400 points drawn the same way. Those points are not shown.

Try three things. Set the separation to 1.2 and place one label on each cluster, then watch the representation rule. Move the separation up to 3.2 and repeat. Then increase the unlabeled points and see whether the representation rule changes at all at 1.2. Expect only a small change: the direction depends on the shape of the clusters, and more points estimate that shape a little more accurately.

A representation from unlabeled data

The book describes semi-supervised learning in deep learning as learning a representation . The feature map is learned so that examples from the same class get similar representations. Points that cluster tightly in the input space should land close together in the representation. A linear classifier in that new space may then generalize better in many cases, so few labels can go further.

(The book writes this relation in the text. The display is added here so that the symbols can be clicked.)

A long-standing version of this idea applies principal components analysis (PCA) before a classifier. PCA finds the direction along which the unlabeled points spread out most. A direction is a vector of length 1, so is a unit vector. Projecting a point onto gives one number, its position along the line.

(This formula is added here and is not printed in the book. It is the average squared distance of the projections from their mean. Here is the mean of the unlabeled inputs, and is how many there are.)

Turn the line and watch the projections. When the line lies across the clusters, the projections spread wide. When it lies along them, the projections bunch up. The spread is largest for one angle, and the readout shows how close your line gets to that largest spread. The separation moves that best angle, and Section 4 returns to that effect.

The unlabeled points set the direction, but the labels do not. Labels enter only when the threshold is placed. This split is what makes the representation useful when labels are scarce.

The two axes use different scales, so the angle on screen differs from the angle in the data. The readout gives the angle in the data.

Sharing parameters with a generative model

The book describes a second route. Build one model in which a generative part, a model of or , and a discriminative part, a model of , share some parameters. The model is then trained on two criteria at once: the supervised criterion and an unsupervised one, the generative criterion.

The weight decides how much of the generative criterion is included. The shared parameters are what link the two parts, so the structure of the inputs pulls on the same numbers that the labels pull on. The book notes that this gives a prior belief: that the structure of is connected to the structure of . A good trade-off can then beat either criterion used alone.

(The book states the two criteria in words, and this line writes their weighted sum with the symbols of this section. The weight is the trade-off the book describes. The sum is added here and is not printed in the book.)

Real generative criteria are log-likelihoods of a neural network, which are hard to draw. The next demo uses a toy with the same structure. Two quadratic curves stand in for the two criteria, and one shared number is chosen to minimize their weighted sum. This is an illustration, not the book’s model.

(These two lines are added here and are not printed in the book. The first curve pulls towards 2 and the second towards −1. The derivative of , set to zero, gives the minimiser.)

At only the supervised pull counts. As grows, the generative pull takes over, and the best moves along the way between the two optima. Predict the value of at before you move the slider.

The conditions for the benefit

The book cites Salakhutdinov and Hinton (2008), who learn the kernel of a kernel machine used for regression. In their work, unlabeled inputs used to model improve considerably. The book also refers to Chapelle et al. (2006) for more information on semi-supervised learning.

The cluster idea has a condition. The unlabeled points help only when the structure they reveal is the structure that separates the labels. The representation rule of Section 2 depends on exactly that. When the two clusters sit close together, the direction of largest spread follows the elongation of each cluster, and the representation mixes the classes.

Predict where the crossover lies before you press a guess. Then compare the two rules across a range of separations.

The sweep uses 200 unlabeled points, and the labels are the first plus point and the first minus point of each sample. The toy is deliberately simple, so its numbers say nothing about real data. They show the shape of the trade-off.

• Semi-supervised learning uses labeled and unlabeled examples together, to estimate .
• A representation learned from unlabeled points can make a few labels go much further.
• PCA finds the direction of largest spread without any labels. It helps only when that direction matches the separation between classes.
• A shared parameter ties a generative criterion to a supervised one, and the weight sets the trade-off between them.
• Unlabeled data is not free help. It helps when the structure it reveals agrees with the labels.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 7.6 (pages 243–244), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.