Elamur Research⌕
☰
home / papers / 2015 / Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Summary

Conventional speech recognizers are assembled from many hand-tuned parts, including feature extractors, acoustic, pronunciation and language models, and speaker adaptation. That makes porting them to a new language or environment slow. Baidu's Deep Speech 2 (DS2) replaces most of this pipeline with one neural network trained with the CTC loss. The network stacks convolutional and recurrent layers and applies Batch Normalization to RNNs. It was trained on 11,940 hours of English and 9,400 hours of Mandarin audio, with synthetic data augmentation. An HPC-style training stack using synchronous SGD across 8–16 GPUs gives a 7x speedup over Deep Speech 1, so a full-scale model trains in days instead of weeks. This made systematic architecture search practical. DS2 cuts English errors by up to 43% relative to its predecessor. On several benchmarks it matches or beats crowdsourced human transcribers, and essentially no language-specific expertise was needed to build it. The authors also show a low-latency production deployment for Mandarin, which makes the case that end-to-end ASR can scale cheaply to real users.

Key points

  • Data and models: training used 11,940 h of English and 9,400 h of Mandarin speech. Models reach up to 11 layers, with many bidirectional recurrent and convolutional layers, and are trained from scratch with CTC without frame-level alignments.
  • Training throughput: training sustains about 50 teraFLOP/s on 16 GPUs, roughly 3 TFLOP/s per GPU or about half of theoretical peak. This relies on a GPU CTC implementation, a custom memory allocator and a custom all-reduce, and brings training time to 3–5 days.
  • Human baseline: the authors measured Amazon Mechanical Turk workers on each benchmark. DS2 beats them on some common test sets and narrows the gap on harder ones.
  • Mandarin deployment: Batch Dispatch gives a 98th-percentile compute latency of 67 ms with 10 concurrent streams. Production uses 16-bit inference, custom small-batch matrix-multiply kernels, and a model with five forward-only recurrent layers (2560 units) plus a row convolution (τ = 19).
  • Accuracy cost of deployment: CER goes from 5.81 for the research system to 6.10 for the deployed one, about 5% relative degradation.

Interesting ideas

  • I1. Build a human baseline yourself. Have human workers transcribe each test set, so "human-level" becomes a measured reference rather than a guess. This works for any Ukrainian NLP or ASR benchmark where annotator quality varies.
  • I2. Choose synchronous SGD for its debuggability. The authors pick synchronous data-parallel SGD over asynchronous parameter servers because it is easier to debug and converges faster at the same parallelism. Fast, reproducible runs speed up research more than raw scale does.
  • I3. Prune beam search by cumulative probability. At each step, keep only the smallest set of candidates whose probabilities add up to p = 0.99, and never more than 40. This sped up Mandarin LM lookups 150x with negligible CER loss. The same trick fits any decoder with a large output vocabulary.
  • I4. Cut inference costs without retraining. Inference is much less sensitive to precision than training, so fp16 weights plus kernels tuned for tiny batches lower serving costs. Batching concurrent user requests on one GPU, as Batch Dispatch does, further trades a little latency for much higher throughput.
  • I5. Prefer language-agnostic pipelines for new languages. One generic end-to-end recipe served both English and Mandarin with little linguistic expertise. This suggests a practical route for languages with fewer engineering resources, such as Ukrainian, if enough labeled audio is collected.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Cited by

192 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 178 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1512.02595 · arXiv 1512.02595

@misc{amodei2015deep,
  title = {{Deep Speech 2: End-to-End Speech Recognition in English and Mandarin}},
  author = {Amodei, Dario and Anubhai, Rishita and Battenberg, Eric and Case, Carl and Casper, Jared and Catanzaro, Bryan and Chen, Jingdong and Chrzanowski, Mike and Coates, Adam and Diamos, Greg and Elsen, Erich and Engel, Jesse and Fan, Linxi and Fougner, Christopher and Han, Tony and Hannun, Awni and Jun, Billy and LeGresley, Patrick and Lin, Libby and Narang, Sharan and Ng, Andrew and Ozair, Sherjil and Prenger, Ryan and Raiman, Jonathan and Satheesh, Sanjeev and Seetapun, David and Sengupta, Shubho and Wang, Yi and Wang, Zhiqian and Wang, Chong and Xiao, Bo and Yogatama, Dani and Zhan, Jun and Zhu, Zhenyao},
  year = {2015},
  doi = {10.48550/arxiv.1512.02595},
  eprint = {1512.02595},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1512.02595}
}

References

70 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. O. Abdel-Hamid, A.-r. Mohamed, H. Jang, and G. Penn. Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition. In ICASSP, 2012. find
  2. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
  3. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end attention-based large vocabulary speech recognition. abs/1508.04395, 2015. http://arxiv.org/abs/1508.04395. — [arXiv:1508.04395](https://arxiv.org/abs/1508.04395) ↗
  4. J. Barker, E. Marxer, Ricard Vincent, and S. Watanabe. The third ’CHiME’ speech separation and recognition challenge: Dataset, task and baselines. 2015. Submitted to IEEE 2015 Automatic Speech Recognition and Understanding Workshop (ASRU). find
  5. S. Baxter. Modern GPU. https://nvlabs.github.io/moderngpu/. ↗
  6. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In International Conference on Machine Learning, 2009. find our card
  7. H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, Norwell, MA, 1993. find
  8. W. Chan, N. Jaitly, Q. Le, and O. Vinyals. http://arxiv.org/abs/1508.01211. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) ↗
  9. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cuDNN: Efficient primitives for deep learning. find
  10. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In USENIX Symposium on Operating Systems Design and Implementation, 2014. find
  11. K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. find
  12. J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio. End-to-end continuous speech recognition using attention-based recurrent nn: First results. abs/1412.1602, 2015. http://arxiv.org/abs/1412.1602. — [arXiv:1412.1602](https://arxiv.org/abs/1412.1602) ↗
  13. C. Cieri, D. Miller, and K. Walker. The Fisher corpus: a resource for the next generations of speech-to-text. In LREC, volume 4, pages 69–71, 2004. find our card
  14. A. Coates, B. Carpenter, C. Case, S. Satheesh, B. Suresh, T. Wang, D. J. Wu, and A. Y. Ng. Text detection and character recognition in scene images with unsupervised feature learning. In International Conference on Document Analysis and Recognition, 2011. find
  15. A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y. Ng, and B. Catanzaro. Deep learning with COTS HPC. In International Conference on Machine Learning, 2013. find
  16. G. Dahl, D. Yu, and L. Deng. Large vocabulary continuous speech recognition with context-dependent DBN-HMMs. In Proc. ICASSP, 2011. find
  17. G. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 2011. find
  18. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, 2012. find
  19. D. Ellis and N. Morgan. Size matters: An empirical study of neural network training for large vocabulary continuous speech recognition. In ICASSP, volume 2, pages 1013–1016. IEEE, 1999. find
  20. E. Elsen. Optimizing RNN performance. http://svail.github.io/rnn_perf. Accessed: 2015-11-24. ↗
  21. M. J. F. Gales, A. Ragni, H. Aldamarki, and C. Gautier. Support vector machines for noise robust ASR. In ASRU, pages 205–2010, 2009. find
  22. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369–376. ACM, 2006. find
  23. A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, 2014. find
  24. A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, 2013. find
  25. H. H. Sak, A. Senior, and F. Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Interspeech, 2014. find
  26. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng. Deep speech: Scaling up end-to-end speech recognition. 1412.5567, 2014. http://arxiv.org/abs/1412.5567. ↗ our card
  27. A. Y. Hannun, A. L. Maas, D. Jurafsky, and A. Y. Ng. First-pass large vocabulary continuous speech recognition using bi-directional recurrent DNNs. abs/1408.2873, 2014. http://arxiv.org/abs/1408.2873. — [arXiv:1408.2873](https://arxiv.org/abs/1408.2873) ↗
  28. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, 8 2013. find our card
  29. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(November):82–97, 2012. find
  30. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735—1780, 1997. find
  31. N. Jaitly and G. Hinton. Vocal tract length perturbation (VTLP) improves speech recognition. In ICML Workshop on Deep Learning for Audio, Speech, and Language Processing, 2013. find
  32. R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In ICML, 2015. Listen, attend, and spell. abs/1508.01211, 2015. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) find
  33. O. Kapralova, J. Alex, E. Weinstein, P. Moreno, and O. Siohan. A big data approach to acoustic model training corpus selection. In Interspeech, 2014. find
  34. K. C. Knowlton. A fast storage allocator. Commun. ACM, 8(10):623–624, Oct. 1965. find
  35. T. Ko, V. Peddinti, D. Povey, and S. Khudanpur. Audio augmentation for speech recognition. In Interspeech, 2015. find
  36. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1106–1114, 2012. find
  37. C. Laurent, G. Pereyra, P. Brakel, Y. Zhang, and Y. Bengio. Batch normalized recurrent neural networks. abs/1510.01378, 2015. http://arxiv.org/abs/1510.01378. — [arXiv:1510.01378](https://arxiv.org/abs/1510.01378) ↗
  38. Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng. Building high-level features using large scale unsupervised learning. In International Conference on Machine Learning, 2012. find
  39. Y. LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, volume 2, pages 97–104, 2004. find
  40. A. Maas, Z. Xie, D. Jurafsky, and A. Ng. Lexicon-free conversational speech recognition with neural networks. In NAACL, 2015. find
  41. Y. Miao, M. Gowayyed, and F. Metz. EESEN: End-to-end speech recognition using deep rnn models and wfst-based decoding. In ASRU, 2015. find
  42. A. Mohamed, G. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, (99), 2011. find
  43. A. S. N. Jaitly, P. Nguyen and V. Vanhoucke. Application of pretrained deep neural networks to large vocabulary speech recognition. In Interspeech, 2012. find
  44. Nervana Systems. Nervana GPU. https://github.com/NervanaSystems/nervanagpu. Accessed: 2015-11-06. ↗
  45. J. Niu, L. Xie, L. Jia, and N. Hu. Context-dependent deep neural networks for commercial mandarin speech recognition applications. In APSIPA, 2013. find
  46. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, 2015. find
  47. R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. abs/1211.5063, 2012. http://arxiv.org/abs/1211.5063. ↗
  48. P. Patarasuk and X. Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. J. Parallel Distrib. Comput., 69(2):117–124, Feb. 2009. find
  49. R. Raina, A. Madhavan, and A. Ng. Large-scale deep unsupervised learning using graphics processors. In 26th International Conference on Machine Learning, 2009. find
  50. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco. Connectionist probability estimators in HMM speech recognition. IEEE Transactions on Speech and Audio Processing, 2(1):161–174, 1994. find
  51. T. Robinson, M. Hochberg, and S. Renals. The use of recurrent neural networks in continuous speech recognition. pages 253–258, 1996. find
  52. T. Sainath, O. Vinyals, A. Senior, and H. Sak. Convolutional, long short-term memory, fully connected deep neural networks. In ICASSP, 2015. find
  53. T. N. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran. Deep convolutional neural networks for LVCSR. In ICASSP, 2013. find
  54. H. Sak, A. Senior, K. Rao, and F. Beaufays. Fast and accurate recurrent neural network acoustic models for speech recognition. abs/1507.06947, 2015. http://arxiv.org/abs/1507.06947. — [arXiv:1507.06947](https://arxiv.org/abs/1507.06947) ↗
  55. H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao. Sequence discriminative distributed training of long shortterm memory recurrent neural networks. In Interspeech, 2014. find
  56. B. Sapp, A. Saxena, and A. Ng. A fast data collection and augmentation procedure for object recognition. In AAAI Twenty-Third Conference on Artificial Intelligence, 2008. find
  57. M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997. find
  58. F. Seide, G. Li, and D. Yu. Conversational speech transcription using context-dependent deep neural networks. In Interspeech, pages 437–440, 2011. find
  59. J. Shan, G. Wu, Z. Hu, X. Tang, M. Jansche, and P. Moreno. Search by voice in mandarin chinese. In Interspeech, 2010. find
  60. H. Soltau, G. Saon, and T. Sainath. Joint training of convolutional and non-convolutional neural networks. In ICASSP, 2014. find
  61. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of momentum and initialization in deep learning. In 30th International Conference on Machine Learning, 2013. find
  62. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. 2014. http://arxiv.org/abs/1409.3215. ↗
  63. C. Szegedy and S. Ioffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. abs/1502.03167, 2015. http://arxiv.org/abs/1502.03167. ↗
  64. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. 2014. find
  65. R. Thakur and R. Rabenseifner. Optimization of collective communication operations in mpich. International Journal of High Performance Computing Applications, 19:49–66, 2005. find
  66. K. Vesely, A. Ghoshal, L. Burget, and D. Povey. Sequence-discriminative training of deep neural networks. In Interspeech, 2013. find
  67. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang. Phoneme recognition using time-delay neural networks,âĂİ acoustics speech and signal processing. IEEE Transactions on Acoustics, Speech and Signal Processing, 37(3):328–339, 1989. find
  68. R. Williams and J. Peng. An efficient gradient-based algorithm for online training of recurrent network trajectories. Neural computation, 2:490–501, 1990. find
  69. T. Yoshioka, N. Ito, M. Delcroix, A. Ogawa, K. Kinoshita, M. F. C. Yu, W. J. Fabian, M. Espi, T. Higuchi, S. Araki, and T. Nakatani. The ntt chime-3 system: Advances in speech enhancement and recognition for mobile multi-microphone devices. In IEEE ASRU, 2015. find
  70. W. Zaremba and I. Sutskever. Learning to execute. abs/1410.4615, 2014. http://arxiv.org/abs/1410.4615. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/