Elamur Research⌕
☰
home / papers / 2017 / Building competitive direct acoustics-to-word models for English conversational speech recognition

Building competitive direct acoustics-to-word models for English conversational speech recognition

Paper · 2017 · Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran et al.

Summary

Direct acoustics-to-word (A2W) models map speech straight to whole words with one neural network. They need no pronunciation lexicon, no decoder and no separately trained language model. Earlier work suggested such models need vastly more audio than phone- or character-based systems, and the authors' own previous A2W system trailed a phone CTC baseline by several WER points on Switchboard. This paper gives a training recipe for a CTC-trained BLSTM word model on the 2000-hour Switchboard+Fisher data. The recipe covers sorting utterances by length, Nesterov momentum, dropout, an output projection layer, initialization from a phone CTC model, and a larger network. With plain peak-picking decoding it reaches 8.8%/13.9% WER on Hub5-2000 SWB/CH. That is competitive with strong sub-word systems that use decoders and LMs. An ablation shows that initialization, data order and regularization contribute most. The authors also propose a spell and recognize (SAR) CTC model, which emits a word's characters before the word itself. This gives readable output for rare or unseen words. The work matters because it shows whole-word recognition is viable at moderate data scale.

Key points

  • Data: a "300-hour" set (262 h of Switchboard-1) and a "2000-hour" set (adding 1698 h of Fisher and 15 h of CallHome). Inputs are 40-dim logMel features with deltas, stacking+decimation and 100-dim speaker i-vectors.
  • Model: a 5-layer BLSTM (320 units per direction) trained with CTC. The vocabulary is words seen at least 5 times, giving 10,000 outputs (300 h) or 25,000 outputs (2000 h). Initialization uses a phone CTC BLSTM and GloVe embeddings for the final layer.
  • On 300 h, ordering utterances shortest-first gave 18.3/28.1 versus 23.0/30.7 for longest-first. The full recipe reaches 14.6/23.6, compared with the earlier 20.8/30.4.
  • On 2000 h, WER fell from 13.0/18.8 to 8.8/13.9. The previous phone CTC system with an LM scored 9.6/16.0.
  • The 25k vocabulary still leaves an OOV rate of 0.5%/0.8% on SWB/CH. This error source is absent in the compared systems, which use larger vocabularies.

Interesting ideas

  • I1. Feeding training sequences from short to long acts as a cheap curriculum: the network settles into a better region early, and length-sorted batches also need less padding.
  • I2. A hard direct model trains far better when it starts from a network trained on an easier, finer-grained task, with output weights seeded from pretrained embeddings.
  • I3. The SAR model emits a spelled-out character form before the whole-word symbol, so a rare or unknown word still yields a readable guess instead of a wrong frequent word.
  • I4. The paper layers recipe ingredients one at a time and reports WER after each addition, which shows clearly where the gain comes from.
  • I5. The authors state their vocabulary's OOV rate next to the error figures, separating coverage failures from modelling failures.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Sort gparse training sentences by ascending token count in early epochs, then measure the gain in LAS against a longest-first control.
  • I2 (tree): Warm-start the gparse encoder from a tagger/morphology model trained on the same treebanks, and seed relation-label output rows from embeddings of the label names.
  • I3 (tokens): Let the lemmatizer output a lemma-class decision plus an explicit character-level fallback for unseen forms, keeping the result inspectable instead of mapping it to a frequent wrong lemma.
  • I4 (learn): Report each parser or repair-rule change as a cumulative ablation table on fixed dev sets, so every accuracy claim can be traced to a specific change.
  • I5 (world): Log the share of entities and verbs missing from the compiled ontology and VerbNet classes alongside answer accuracy, so knowledge gaps are not blamed on the reader logic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 worksSpeech & spoken language · 233 works

Cite

Cite as Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran, George Saon and Michael Picheny. 2017. Building competitive direct acoustics-to-word models for English conversational speech recognition. arXiv:1712.03133.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1712.03133 · arXiv 1712.03133

@misc{audhkhasi2017building,
  title = {{Building competitive direct acoustics-to-word models for English conversational speech recognition}},
  author = {Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran, George Saon and Michael Picheny},
  year = {2017},
  doi = {10.48550/arxiv.1712.03133},
  eprint = {1712.03133},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1712.03133}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. F. Jelinek, Statistical methods for speech recognition, MIT press, 1997. find
  2. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012. ↗
  3. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, 2006, pp. 369–376. find
  4. A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks.,” in ICML, 2014, vol. 14, pp. 1764–1772. find
  5. G. Zweig, C. Yu, J. Droppo, and A. Stolcke, “Advances in all-neural speech recognition,” in Proc. ICASSP, 2017, pp. 4805– 4809. find
  6. H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” in Proc. Interspeech, 2015. find
  7. Y. Miao, M. Gowayyed, and F. Metze, “EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,” in Proc. ASRU, 2015, pp. 167–174. find
  8. Y. Miao, M. Gowayyed, X. Na, T. Ko, F. Metze, and A. Waibel, “An empirical exploration of CTC acoustic models,” in Proc. ICASSP, 2016, pp. 2623–2627. ↗
  9. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “Deep speech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.5567, 2014. ↗
  10. D. Amodei et al., “Deep speech 2: End-to-end speech recognition in English and Mandarin,” in International Conference on Machine Learning, 2016, pp. 173–182. ↗
  11. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Proc. ICASSP, 2016, pp. 4945–4949. find
  12. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in Proc. ICLR, 2015. ↗
  13. A. L. Maas, Z. Xie, D. Jurafsky, and A. Y. Ng, “Lexicon-free conversational speech recognition with neural networks,” in Proc. HLT-NAACL, 2015, pp. 345–354. find
  14. W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Proc. ICASSP, 2016, pp. 4960–4964. find
  15. W. Chan, Y. Zhang, Q. Le, and N. Jaitly, “Latent sequence decompositions,” arXiv preprint arXiv:1610.03035, 2016. ↗
  16. H. Soltau, H. Liao, and H. Sak, “Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition,” in Proc. Interspeech, 2017. find
  17. K. Audhkhasi, B. Ramabhadran, G. Saon, M. Picheny, and D. Nahamoo, “Direct acoustics-to-word models for English conversational speech recognition,” in Proc. Interspeech, 2017, pp. 959–963.
  18. G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” in Proc. Interspeech, 2017.
  19. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global Vectors for Word Representation,” in Proc. EMNLP, 2014, vol. 14, pp. 1532–1543.
  20. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. AISTATS, 2010, pp. 249–256. find
  21. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proc. ICML, 2009.
  22. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, “Low-rank matrix factorization for deep neural network training with high-dimensional output targets,” in Proc. ICASSP, 2013, pp. 6655–6659. find
  23. D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI,” in Proc. Interspeech, 2016, pp. 2751–2755. find
  24. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Achieving human parity in conversational speech recognition,” arXiv preprint arXiv:1610.05256, 2016. ↗
  25. T. Sercu and V. Goel, “Dense prediction on sequences with time-dilated convolutions for speech recognition,” arXiv preprint arXiv:1611.09288, 2016. ↗
  26. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “The Microsoft 2016 conversational speech recognition system,” in Proc. ICASSP, 2017, pp. 5255– 5259. ↗
  27. H. Liu, Z. Zhu, X. Li, and S. Satheesh, “Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling,” in Proc. ICML, 2017. ↗
  28. E. Battenberg, J. Chen, R. Child, A. Coates, Y. Gaur, Y. Li, H. Liu, S. Satheesh, D. Seetapun, A. Sriram, and Z. Zhu, “Exploring neural transducers for end-to-end speech recognition,” arXiv preprint arXiv:1707.07413, 2017. ↗
  29. H. Sak, M. Shannon, K. Rao, and F. Beaufays, “Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping,” Proc. Interspeech 2017, pp. 1298–1302, 2017. find
  30. O. Siohan and M. Bacchiani, “Fast vocabulary-independent audio search using path-based graph indexing,” in Proc. European Conference on Speech Communication and Technology, 2005. find
  31. B. Ramabhadran, A. Sethy, J. Mamou, B. Kingsbury, and U. Chaudhari, “Fast decoding for open vocabulary spoken term detection,” in Proc. NAACL-HLT, 2009, pp. 277–280. find
  32. A. Rastrow, A. Sethy, B. Ramabhadran, and F. Jelinek, “Towards using hybrid word and fragment units for vocabulary independent LVCSR systems,” in Interspeech, 2009, pp. 1931– 1934. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/audhkhasi-2017-building-competitive-direct-acoustics-word/