Elamur Research⌕
☰
home / papers / 2017 / Direct Acoustics-to-Word Models for English Conversational Speech Recognition

Direct Acoustics-to-Word Models for English Conversational Speech Recognition

Paper · 2017 · Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon et al.

Summary

The paper asks whether one neural network can turn speech directly into words, with no phone inventory, pronunciation dictionary, language model or decoder. It also asks whether this works on standard public benchmarks with moderate data, not only with very large private collections. The authors train CTC models whose output units are whole words. They use 300 hours of Switchboard and about 2000 hours of Switchboard+Fisher, with a 5-layer BLSTM over logMel and i-vector features. From random weights, the word model failed to converge. It became trainable once the BLSTM was initialised from a phone CTC model, and GloVe embeddings used to set the final layer helped further. A hierarchical multi-task CTC variant beat random initialisation but lost to plain phone-model initialisation. With no LM or decoder, the 2000-hour word model reaches 13.0%/18.8% WER on Hub5-2000 Switchboard/CallHome. A phone CTC system with a 4-gram LM scores 9.6%/16.0% on the same sets. The result shows that word-level end-to-end recognition is feasible on public data, and decoding becomes a simple per-frame pick followed by collapsing.

Key points

  • Word vocabulary keeps only words with at least 5 acoustic examples: about 10k words for 300 h and about 25k for 2000 h. Training OOV rates are 1.5% and 0.5%; one shared OOV token is used and mapped to silence at scoring time.
  • The phone CTC baseline uses 44 phones plus blank. The 2000-hour version reaches 10.2/16.5 (small LM) and 9.6/16.0 (big LM). A hybrid BLSTM gets 7.7/14.0, but phone CTC decodes about 2x faster with 8x less memory.
  • The input pipeline stacks two logMel frames and drops every other frame, halving sequence length at no accuracy cost. 100-dim i-vectors are appended, giving 180-dim inputs. Training uses Torch, cuDNN and Warp-CTC.
  • Random initialisation of the word CTC model did not converge despite tuning. Copying the whole phone CTC BLSTM fixed this, and it beat hierarchical CTC at any tested phone/word loss weight α.
  • The word CTC system makes no more errors than phone CTC on 72.1%/74.9% of test utterances. The paper also reports LM lattice rescoring of the word model.

Interesting ideas

  • I1. Train a model on fine-grained, easy units first, then reuse its whole encoder for a coarse task with a huge label set that cannot be learned from scratch.
  • I2. Seed the rows of a very large output layer with pretrained embeddings, so rare labels inherit co-occurrence knowledge instead of starting as noise.
  • I3. Cap the label inventory by a minimum-frequency threshold, collapse everything else into one explicit OOV class, and report the resulting OOV rate.
  • I4. Compare two systems per item, by the share of utterances where one makes no more errors than the other, rather than only by an aggregate error rate.
  • I5. Test whether multi-task auxiliary losses actually beat simple initialisation; here a joint phone+word loss lost to plain weight copying.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Initialise gparse's encoder from a tagger/morphology model trained on the same treebanks, so dependency training starts from features that already detect word-level structure.
  • I2 (tokens): A lemma or fine-tag classifier in crate en could have its output vectors initialised from embeddings, helping rare lemmas with few training examples.
  • I3 (world): When compiling verb classes or ontology labels from WordNet/FrameNet/Wikidata, an explicit "unknown" bucket with a measured coverage rate keeps the knowledge layer honest about what it cannot name.
  • I4 (learn): When comparing the perceptron parser and gparse, or rules before and after self-induced repair, a per-sentence "no worse than" share shows where a change helps or hurts.
  • I5 (learn): Before adding joint objectives to teacher-tier training, run the plain warm-start baseline as a control, since it may already be better.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 worksSpeech & spoken language · 233 works

Cite

Cite as Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon, Michael Picheny and David Nahamoo. 2017. Direct Acoustics-to-Word Models for English Conversational Speech Recognition. arXiv:1703.07754.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.07754 · arXiv 1703.07754

@misc{audhkhasi2017direct,
  title = {{Direct Acoustics-to-Word Models for English Conversational Speech Recognition}},
  author = {Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon, Michael Picheny and David Nahamoo},
  year = {2017},
  doi = {10.48550/arxiv.1703.07754},
  eprint = {1703.07754},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.07754}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012. ↗
  2. G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” arXiv preprint arXiv:1703.02136, 2017. ↗
  3. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “The Microsoft 2016 conversational speech recognition system,” in Proc. ICASSP, 2017, pp. 5255–5259. ↗
  4. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Černockỳ, “Empirical evaluation and combination of advanced language modeling techniques,” in Proc. Interspeech, 2011. find
  5. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, “One billion word benchmark for measuring progress in statistical language modeling,” in Proc. Interspeech, 2014.
  6. S. F. Chen, “Shrinking exponential language models,” in Proc. HLT-NAACL, 2009, pp. 468–476. find
  7. F. Jelinek, Statistical methods for speech recognition. MIT press, 1997. find
  8. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, 2006, pp. 369–376. find
  9. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014. ↗
  10. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Proc. ICASSP, 2016, pp. 4945–4949. find
  11. A. L. Maas, Z. Xie, D. Jurafsky, and A. Y. Ng, “Lexicon-free conversational speech recognition with neural networks,” in Proc. HLT-NAACL, 2015, pp. 345–354. find
  12. A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks.” in ICML, vol. 14, 2014, pp. 1764–1772. find
  13. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “Deep speech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.5567, 2014. ↗
  14. Y. Miao, M. Gowayyed, and F. Metze, “EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,” in Proc. ASRU, 2015, pp. 167–174. find
  15. Y. Miao, M. Gowayyed, X. Na, T. Ko, F. Metze, and A. Waibel, “An empirical exploration of CTC acoustic models,” in Proc. ICASSP, 2016, pp. 2623–2627. ↗
  16. H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” in Proc. Interspeech, 2015. find
  17. G. Zweig, C. Yu, J. Droppo, and A. Stolcke, “Advances in all-neural speech recognition,” in Proc. ICASSP, 2017, pp. 4805– 4809. find
  18. H. Soltau, H. Liao, and H. Sak, “Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition,” arXiv preprint arXiv:1610.09975, 2016. ↗
  19. S. Fernandez, A. Graves, and J. Schmidhuber, “Sequence labelling in structured domains with hierarchical recurrent neural networks,” in Proc. IJCAI, 2007, pp. 774–779. find
  20. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global Vectors for Word Representation,” in Proc. EMNLP, vol. 14, 2014, pp. 1532–1543.
  21. N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011. ↗
  22. R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A Matlab-like Environment for Machine Learning,” in Proc. BigLearn, NIPS Workshop, 2011. find
  23. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “CUDNN: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014. ↗
  24. K. Rao and H. Sak, “Multi-accent speech recognition with hierarchical grapheme based models,” in Proc. ICASSP, 2017, pp. 4815–4819. find
  25. K. Audhkhasi, A. Sethy, and B. Ramabhadran, “Semantic word embedding neural network language models for automatic speech recognition,” in Proc. ICASSP, 2016, pp. 5995–5999. find
  26. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/audhkhasi-2017-direct-acoustics-word-models-english/