Speech Recognition

From sound to frames

Speech recognition turns an acoustic signal into the words the speaker meant. The input is a sequence of short frames. Traditionally each frame is 20 milliseconds of audio. The book writes the input as , a sequence of acoustic vectors , and the output as , a sequence of words or characters.

Before any model runs, the audio is cut into frames. Predict first.

The number of frames in one second follows from the frame length and the step. The formula is added here. The book gives the 20 millisecond frame in words and does not print this count.

The most probable words

The book defines the ideal recognizer as the function that returns the most probable word sequence given the audio. The equation below is equation (12.4). The true distribution is unknown, so systems estimate it from data. argmax, short for argument of the maximum, is the input that makes the expression largest. Here it returns the word sequence, not its score.

The book keeps the argmax in one piece. The toy below splits the score into two parts, a sound score and a language score, which is one common way of writing the same choice. It uses a weight for the sound. The scores are made up for this page and are added here.

The toy shows the trade-off. With a small weight the language score decides, and the phrase that fits the sound best but is less plausible loses. With a large weight the sound decides. Bayes’ rule says that the probability of the words given the sound is proportional to the probability of the sound given the words, times the probability of the words. The formula below is added here. It is that rule in logarithms, with the acoustic term raised to the power .

From GMM-HMM to deep networks

For decades speech systems used a GMM-HMM pipeline, built from a hidden Markov model (HMM) and a Gaussian mixture model (GMM). A phoneme is one of the basic speech sounds of a language. An HMM generated a sequence of phonemes and sub-phonemic states, such as the beginning, middle and end of a phoneme, and a GMM turned each state into a short piece of waveform. The book notes that neural networks were tried in the late 1980s and early 1990s, and matched the GMM-HMM systems of their time. Industry did not see a compelling reason to switch, because the engineering invested in GMM-HMM systems was large. Until the late 2000s most neural network work on speech learned extra features for GMM-HMM systems.

From about 2009, deep networks were trained with unsupervised pretraining, in which each layer is first trained as a restricted Boltzmann machine (RBM) on the input without labels. The trained network then replaced the GMM part. Predict first.

The book then describes the next steps. Convolutional networks replicate weights across time and frequency, where earlier time-delay networks replicated them across time only. They treat the spectrogram as an image with time on one axis and frequency on the other. Rectified linear units and dropout later took the place of pretraining as the main training tools. End-to-end systems go further and remove the HMM, so that a deep recurrent network learns the alignment itself. The book says that some groups found the unsupervised pretraining was unnecessary once they had large labeled datasets.

Milestones and end-to-end systems

The book gives a few phoneme error rates on the TIMIT corpus, which has 39 phonemes to tell apart. TIMIT has been the benchmark for phoneme recognition since then, a role similar to that of MNIST for object recognition. In 1991 the neural result was better than or comparable to HMM-based systems. The numbers show the size of the change. The demo below reveals them in order, and the bars are the book’s numbers. Each bar is a single result from a single paper.

The toy shows the numbers. The book adds that the change was dramatic. Word error rates improved by around 30% in the years after 2009, after a long period of little change with GMM-HMM systems. End-to-end systems with deep recurrent networks and connectionist temporal classification (CTC) are the book’s current direction, and they replace the HMM as well. The book also mentions speaker-adaptive features, which reduced the error further, and the extension from phonemes to large-vocabulary recognition.

• Speech recognition maps audio, cut into frames of about 20 milliseconds, to the most probable word sequence.
• The argmax in equation (12.4) is the ideal goal. Real systems weigh the sound against the language, and the weight decides which transcription wins.
• Deep networks first replaced the GMM that scored acoustic features for each phoneme state, and the HMM stayed for a while.
• On TIMIT the phoneme error rate fell from 26% in 1991 to 17.7% in the 2013 deep recurrent system the book cites.

Check yourself

Questions that mix the whole module. Each one is scored, and the score feeds your mastery of this module.

Source: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, Section 12.3 (pages 458–460), free to read at deeplearningbook.org. Where the book prints equation numbers, the numbers here follow the book. The order of ideas follows the book. The explanations and all demos were written for this page.