Elamur Research
☰
home / papers / 2015 / Attention-Based Models for Speech Recognition

Attention-Based Models for Speech Recognition

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1506.07503

Summary

Attention-based sequence generators had worked well for translation and handwriting synthesis, but speech is harder. Inputs span thousands of noisy frames, and similar sounds recur within one recording, so attention driven only by content can lose track of where it is. Chorowski, Bahdanau, Serdyuk, Cho and Bengio study phoneme recognition on TIMIT with an attention-based recurrent sequence generator (ARSG) that reads a deep BiRNN encoding of acoustic features.

A direct adaptation of the Bahdanau et al. translation model does well on the standard test set. It collapses on long inputs, however, apparently because it learns to track absolute position. The authors fix this with hybrid attention: they convolve the previous alignment with learned filters and feed the resulting features into the scoring function alongside content. They also add windowed decoding and a smoother normalization that discourages focusing on a single frame.

The resulting fully neural recognizer matches conventional systems and handles utterances far longer than those seen in training. The location-aware principle is generic and could be applied to other attention models.

Key points

  • Baseline (content-only attention from machine translation): 18.7% PER on TIMIT. On artificially lengthened utterances it aligns roughly the first 40 phones, then jumps to the end of the recording.
  • Adding convolutional location features lowers PER to 18.0%. The model stays near 20% PER on inputs built from up to about 10–11 repeated or concatenated test utterances.
  • The smoothed score normalization further reduces PER to 17.6%.
  • Windowing around the median of the previous alignment cuts attention cost from O(LT) to O(L+T). It was used only at decoding time, and a wide window (±150 frames) beat narrower ones.
  • The experiments used Theano, PyLearn2 and Blocks. An earlier version appeared at the NIPS 2014 Deep Learning Workshop.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 120 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.07503 · arXiv 1506.07503

@misc{chorowski2015attention,
  title = {{Attention-Based Models for Speech Recognition}},
  author = {Chorowski, Jan and Bahdanau, Dzmitry and Serdyuk, Dmitriy and Cho, Kyunghyun and Bengio, Yoshua},
  year = {2015},
  doi = {10.48550/arxiv.1506.07503},
  eprint = {1506.07503},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.07503}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850, August 2013. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, September 2014. find our card
  3. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv:1502.03044, February 2015. find our card
  4. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204–2212, 2014. find
  5. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end continuous speech recognition using attention-based recurrent NN: First results. arXiv:1412.1602 [cs, stat], December 2014. — [arXiv:1412.1602](https://arxiv.org/abs/1412.1602) find
  6. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.5401, 2014. find
  7. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv:1410.3916, 2014. find
  8. Mark Gales and Steve Young. The application of hidden markov models in speech recognition. Found. Trends Signal Process., 1(3):195–304, January 2007. find
  9. G. Hinton, Li Deng, Dong Yu, G.E. Dahl, A Mohamed, N. Jaitly, A Senior, V. Vanhoucke, P. Nguyen, T.N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97, November 2012. find
  10. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deepspeech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567, 2014. find our card
  11. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural. Comput., 9(8):1735–1780, 1997. find
  12. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP 2014, October 2014. to appear. find
  13. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML-06, 2006. find
  14. Alex Graves. Sequence transduction with recurrent neural networks. In ICML-12, 2012. find our card
  15. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient based learning applied to document recognition. Proc. IEEE, 1998. find
  16. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ICASSP 2013, pages 6645–6649. IEEE, 2013. find
  17. Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML-14, pages 1764–1772, 2014. find
  18. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. Weakly supervised memory networks. arXiv preprint arXiv:1503.08895, 2015. find
  19. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. DARPA TIMIT acoustic phonetic continuous speech corpus, 1993. find
  20. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, and others. The kaldi speech recognition toolkit. In Proc. ASRU, pages 1–4, 2011. find
  21. Matthew D Zeiler. ADADELTA: An adaptive learning rate method. arXiv:1212.5701, 2012. find our card
  22. Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348–2356. Curran Associates, Inc., 2011. find
  23. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. find
  24. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014. find
  25. László Tóth. Combining time-and frequency-domain convolution in convolutional neural network-based phone recognition. In ICASSP 2014, pages 190–194, 2014. find
  26. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535, 2015. find our card
  27. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral… find
  28. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012. find
  29. Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Frédéric Bastien, and Yoshua Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013. — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) find
  30. Bart van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-Farley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning. arXiv:1506.00619 [cs, stat], June 2015. find