Section 1
Why pooling is costly in an energy model
Convolution lets a Boltzmann machine handle images. Replacing the matrix product by a small convolution kernel keeps the number of weights manageable and shares them across positions, and Desjardins and Bengio (2008) showed that this works well for RBMs. Deep convolutional networks also use pooling to shrink each layer, usually with the maximum of the pooled units. Energy models have no obvious version of max pooling, and the naive one is expensive.
detector units
pooling unit
detectors in one region
Predict firstFor a 3 by 3 pooling region with nine detector units, how many energy configurations does the naive max pooling constraint need to evaluate?
How to use this pageClick any colored symbol in an equation or a sentence and it explains itself. Move the controls before you read the text under each demo, and try the question before the reveal.
Configurations to evaluate, naive against probabilistic
| Region | Detectors | Naive configurations | Probabilistic states |
|---|
Challenge +8 XPSet the region to 4 by 4 so that the naive constraint needs at least 60,000 evaluations for one pooling unit.
Lee et al. (2009) made the pooling constraint computable. Before that, a pooling unit over binary detectors enforces by setting the energy to infinity when the constraint fails. This needs configurations to compute the normalizer. For a 3 by 3 region that is 512 evaluations per pooling unit, and the cost doubles with every added detector.
Click any coloured symbol to see what it means.
Probabilistic max pooling
The strategy is to constrain the detectors so that at most one of them is active at a time. A region of detectors then has total states: one for each detector being on, and one state in which all detectors are off. The pooling unit is on exactly when one of the detectors is on, and the all-off state has energy zero. The model keeps only these states, so the sum over configurations is short. It is a model with one variable that has states, or equivalently one that gives infinite energy to every other joint assignment.
Click any coloured symbol to see what it means.
One active detector per region
Click a detector to make it the only one on. Click the active detector to switch the region to all off. Only the states the model keeps are reachable.
Challenge +8 XPActivate one detector, then click a different detector in the same region.
Predict firstHow many joint states does probabilistic max pooling keep for n detectors in one region?
The price is that the detectors become mutually exclusive. The constraint can be a useful regularizer, or it can limit capacity. The model also cannot use overlapping pooling regions, which feedforward networks usually need for their best results, so this constraint probably greatly reduces the performance of convolutional Boltzmann machines.
Section 3
Overlapping pooling regions
The one-active-detector rule applies within each region. When two regions share a detector, that detector is part of both constraints at once. The demo uses two regions of three detectors that overlap on the middle one. Clicking a detector applies the rule to every region that contains it. The book states that probabilistic max pooling does not support overlapping regions, so this is a what-if, and it shows what the rule would do if regions were allowed to share a detector.
Two regions that share a detector
Region A covers detectors 1 to 3, and region B covers detectors 3 to 5. Detector 3 belongs to both.
Challenge +8 XPTurn on the shared detector 3 so that both pooling units are on.
A shared detector can turn on both pooling units, but it must then turn off every other detector in both regions. The overlap therefore couples the two pooling units and restricts the joint states more than either region alone. The book names this as a real cost. Overlapping regions are usually needed for the best feedforward results, and the method does not support them.
Section 4
Boundaries and input size
Connections in a Boltzmann machine are symmetric, so the boundary pixels matter. Without zero padding, there are fewer hidden units than visible units, and the edge pixels lie in the receptive field of fewer hidden units. With zero padding, the hidden units at the boundary are driven by fewer real pixels and may fail to turn on when needed. The demo counts how many hidden units see each visible pixel in a one-dimensional row with a kernel of width three.
How many hidden units see each pixel
Predict firstWith no zero padding, how many hidden units cover the first visible pixel of a width three kernel?
Input size is the second problem. The partition function changes when the input size changes. Convolutional networks cope by enlarging their pooling regions with the input, but large Boltzmann machine pooling regions become too expensive for the naive approach, and the one-active constraint does not vary its region size easily. Suppose a model learns 2 by 2 pooling over edge detectors. If the image grows by 50% in each direction and the region grows to 3 by 3, the constraint allows each edge to appear only once in a 3 by 3 region, so the model produces edges with less density. A model with probabilistic max pooling can still accept images of different sizes, as long as its output is a feature map that scales with the image.
What to take away
- Convolution makes Boltzmann machines practical for images, but naive max pooling needs 2 to the power of n evaluations for n detectors.
- Probabilistic max pooling keeps n + 1 states, with at most one detector on, and that rule blocks overlapping regions.
- Without padding, boundary pixels are covered by fewer hidden units. With padding, boundary hidden units see fewer real pixels and may fail to activate.
- Changing the input size is hard. The partition function changes, and enlarged pooling regions make the one-active constraint thin out the edges.
Final quizCheck yourself
Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.
Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 20.6 (pages 683–685), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.