Elamur Research
☰
home / papers / 2014 / Deep Speech: Scaling up end-to-end speech recognition

Deep Speech: Scaling up end-to-end speech recognition

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1412.5567

Summary

Conventional speech recognizers chain together many hand-tuned components, including engineered features, acoustic models, HMMs and phoneme dictionaries. They also tend to degrade badly in noise. Baidu's Deep Speech replaces most of this chain with one recurrent network that converts spectrograms straight into English characters. It is trained with the CTC loss, so transcripts need no prior alignment to the audio. The architecture is deliberately plain so that it runs efficiently across multiple GPUs: three feed-forward layers with clipped ReLU, one bidirectional recurrent layer without LSTM gates, then a further dense layer and a softmax. The authors gathered thousands of hours of speech and synthesized extra examples containing noise and the Lombard effect, so the model could learn robustness from data alone. At decoding time, an N-gram language model is combined with the network through beam search. The system reached 16.0% error on the full Switchboard Hub5'00 test set, the best published figure at the time. It also beat commercial APIs on noisy audio. The work showed that scale in data and compute could rival elaborate engineered pipelines.

Key points

  • Model and training: The 5-layer RNN has roughly 5 billion connections when unrolled over a typical utterance. Training uses CTC loss, Nesterov momentum of 0.99, and 5–10% dropout applied only to the non-recurrent layers. Test-time accuracy is improved by averaging over ensembles and over audio shifted by 5 ms.
  • Language model: A KenLM N-gram model was built from 220 million phrases with a vocabulary of 495,000 words. Decoding uses beams of 1000–8000.
  • Switchboard result: Deep Speech reached 16.0% WER on the full Hub5'00 test set.
  • Noisy speech result: On the authors' own noisy test set, Deep Speech scored 19.06% WER. The strongest commercial competitor, Google API, scored 30.47%. The other systems compared were Apple Dictation, Bing Speech and wit.ai.
  • Effect of noise synthesis: Adding synthesized noise to 5000 hours of training data cut noisy-utterance WER from 28.7% to 22.6%. Clean-speech accuracy stayed about the same.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

137 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 133 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1412.5567 · arXiv 1412.5567

@misc{hannun2014deep,
  title = {{Deep Speech: Scaling up end-to-end speech recognition}},
  author = {Hannun, Awni and Case, Carl and Casper, Jared and Catanzaro, Bryan and Diamos, Greg and Elsen, Erich and Prenger, Ryan and Satheesh, Sanjeev and Sengupta, Shubho and Coates, Adam and Ng, Andrew Y.},
  year = {2014},
  doi = {10.48550/arxiv.1412.5567},
  eprint = {1412.5567},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.5567}
}

References

44 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, Norwell, MA, 1993. find
  2. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cuDNN: Efficient primitives for deep learning. find
  3. C. Cieri, D. Miller, and K. Walker. The Fisher corpus: a resource for the next generations of speech-to-text. In LREC, volume 4, pages 69–71, 2004. find
  4. D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber. Flexible, high performance convolutional neural networks for image classification. In International Joint Conference on Artificial Intelligence, pages 1237–1242, 2011. find
  5. D. C. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In Computer Vision and Pattern Recognition, pages 3642–3649, 2012. find
  6. A. Coates, B. Carpenter, C. Case, S. Satheesh, B. Suresh, T. Wang, D. J. Wu, and A. Y. Ng. Text detection and character recognition in scene images with unsupervised feature learning. In International Conference on Document Analysis and Recognition, 2011. find
  7. A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y. Ng, and B. Catanzaro. Deep learning with COTS HPC. In International Conference on Machine Learning, 2013. find
  8. A. Coates, H. Lee, and A. Y. Ng. An analysis of single-layer networks in unsupervised feature learning. In 14th International Conference on AI and Statistics, pages 215–223, 2011. find
  9. G. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 2011. find
  10. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, 2012. find
  11. D. Ellis and N. Morgan. Size matters: An empirical study of neural network training for large vocabulary continuous speech recognition. In ICASSP, volume 2, pages 1013–1016. IEEE, 1999. find
  12. X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In 14th International Conference on Artificial Intelligence and Statistics, pages 315–323, 2011. find
  13. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369– 376. ACM, 2006. find
  14. A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, 2014. find
  15. R. Grosse, R. Raina, H. Kwong, and A. Y. Ng. Shift-invariance sparse coding for audio classification. arXiv preprint arXiv:1206.5241, 2012. — [arXiv:1206.5241](https://arxiv.org/abs/1206.5241) find
  16. A. Y. Hannun, A. L. Maas, D. Jurafsky, and A. Y. Ng. First-pass large vocabulary continuous speech recognition using bi-directional recurrent DNNs. abs/1408.2873, 2014. http://arxiv.org/abs/1408.2873. — [arXiv:1408.2873](https://arxiv.org/abs/1408.2873) ↗
  17. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, 8 2013. find our card
  18. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(November):82–97, 2012. find
  19. G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. abs/1406.7806, 2014. http://arxiv.org/abs/1406.7806. ↗
  20. J.-C. Junqua. The Lombard reflex and its role on human listeners and automatic speech recognizers. Journal of the Acoustical Society of America, 1:510–524, 1993. find
  21. D. R. Kincaid, T. C. Oppe, and D. M. Young. Itpackv 2d users guide. 1989. find
  22. B. Kingsbury, T. Sainath, and H. Soltau. Scalable minimum Bayes risk training of deep neural network acoustic models using distributed hessian-free optimization. In Interspeech, 2012. find
  23. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1106–1114, 2012. find
  24. Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng. Building high-level features using large scale unsupervised learning. In International Conference on Machine Learning, 2012. find
  25. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541– 551, 1989. find
  26. Y. LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, volume 2, pages 97–104, 2004. find
  27. H. Lee, P. Pham, Y. Largman, and A. Y. Ng. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems, pages 1096–1104, 2009. find
  28. A. L. Maas, A. Y. Hannun, C. T. Lengerich, P. Qi, D. Jurafsky, and A. Y. Ng. Increasing deep neural network acoustic model size for large vocabulary continuous speech recognition. abs/1406.7806, 2014. http://arxiv.org/abs/1406.7806. — [arXiv:1406.7806](https://arxiv.org/abs/1406.7806) ↗
  29. A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech, and Language Processing, 2013. find
  30. A. Mohamed, G. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, (99), 2011. find
  31. V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In 27th International Conference on Machine Learning, pages 807–814, 2010. find
  32. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, K. Veselý, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, and G. Stemmer. The Kaldi speech recognition toolkit. In ASRU, 2011. find
  33. R. Raina, A. Madhavan, and A. Ng. Large-scale deep unsupervised learning using graphics processors. In 26th International Conference on Machine Learning, 2009. find
  34. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco. Connectionist probability estimators in HMM speech recognition. IEEE Transactions on Speech and Audio Processing, 2(1):161–174, 1994. find
  35. T. Sainath, B. Kingsbury, A. Mohamed, G. Dahl, G. Saon, H. Soltau, T. Beran, A. Aravkin, and B. Ramabhadran. Improvements to deep convolutional neural networks for LVCSR. In ASRU, 2013. find
  36. T. N. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran. Deep convolutional neural networks for LVCSR. In ICASSP, 2013. find
  37. B. Sapp, A. Saxena, and A. Y. Ng. A fast data collection and augmentation procedure for object recognition. In AAAI Twenty-Third Conference on Artificial Intelligence, 2008. find
  38. M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997. find
  39. F. Seide, G. Li, X. Chen, and D. Yu. Feature engineering in context-dependent deep neural networks for conversational speech transcription. In ASRU, 2011. find
  40. H. Soltau, G. Saon, and T. N. Sainath. Joint training of convolutional and non-convolutional neural networks. In ICASSP, 2014. find
  41. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of momentum and initialization in deep learning. In 30th International Conference on Machine Learning, 2013. find
  42. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. 2014. http://arxiv.org/abs/1409.3215. ↗
  43. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. 2014. find
  44. K. Vesely, A. Ghoshal, L. Burget, and D. Povey. Sequence-discriminative training of deep neural networks. In Interspeech, 2013. find