Elamur Research⌕
☰
home / papers / 2021 / On the limit of English conversational speech recognition

On the limit of English conversational speech recognition

Paper · 2021 · Zoltan Tuske, George Saon and Brian Kingsbury

Summary

The paper asks how much further attention-based encoder-decoder ASR can be pushed on the Switchboard conversational telephone benchmark. The authors start from their earlier heavily regularized single-head LSTM encoder-decoder system. They then add several improvements:

  • the AdamW optimizer in place of SGD;
  • i-vector speaker embeddings;
  • a high-time-resolution log-Mel front end with two spectral augmentations (small-energy masking and a new high energy clipping);
  • conformer encoders and decoders;
  • cross-utterance transformer-XL language models;
  • a decoding rule that log-linearly combines several acoustic models and LMs.

The decoder also corrects for the language prior that the end-to-end model has already absorbed, using a density-ratio term with negative weight. On Switchboard-300, the new training ingredients give about 4% relative improvement. Fusing an LSTM model with an LSTM conformer and a stronger LM sets a record of 5.0% WER on SWB and 10.0% on CHM of Hub5'00. Similar gains on Switchboard-2000 yield a new state of the art, which the authors argue is close to the ceiling of this benchmark.

Key points

  • On Switchboard-300 (LSTM model), Hub5'00 total WER without an external LM falls from 11.2% with SGD to 10.4% with AdamW, and to 10.2% when i-vectors are added. With shallow-fusion LM, it falls from 9.8% to 9.4% to 9.3%.
  • With simple LSTM models, the probability-ratio LM integration reaches 5.9% WER on SWB and 11.5% on CHM.
  • The alternative features:
    • use a 2.5 ms frame shift with a 10 ms window, 20 Mel filters and 7th-root compression;
    • apply SEM with 10% probability;
    • apply HEC 40% of the time, clipping each channel at a random 80–100th percentile and then rescaling energy.
  • The output units are BPE with 600 rules (300 h) and 1000 rules (2000 h). The external LMs are trained on the 24M-word Switchboard+Fisher text: a 2-layer, 2048-unit LSTM and a 10-layer transformer-XL.
  • Used alone, conformer models perform about the same as LSTMs. Combining the two architectures and decoding with a better LM gives the record 5.0% SWB and 10.0% CHM WER on Switchboard-300.

Interesting ideas

  • I1. Before adding an external LM, the decoder divides out a separately estimated model of the end-to-end system's internal language prior, so the same linguistic knowledge is not counted twice.
  • I2. The final decision is one weighted log-linear sum of several LMs and several acoustic models, plus explicit reward terms for hypothesis length and for attention covering the whole input.
  • I3. Speed- and tempo-perturbed copies of a recording are given their own new speaker identity, so the augmentation does not contradict the speaker conditioning vector.
  • I4. Gains are reported both with and without the external LM, which shows that improvements shrink once strong external knowledge is fused in.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When gparse scores are fused with an external selectional-preference or lexical model, we could subtract a prior estimated from the parser's own training treebank, so attachment knowledge already learned by the parser is not double-weighted.
  • I2 (learn): Silver-data agreement among witness parsers could use tuned per-parser log-linear weights instead of plain voting, with an added coverage bonus for analyses that attach every token, making each witness's contribution explicit.
  • I3 (learn): When we create paraphrased or perturbed training texts, they should be tagged as distinct sources, so that source- or genre-level features do not leak from the originals and skew evaluation.
  • I4 (learn): Each new component of the pipeline should be evaluated both with and without the compiled knowledge layer, to check that its benefit is not already supplied by the knowledge graph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 worksSpeech & spoken language · 233 works

Cite

Cite as Zoltan Tuske, George Saon and Brian Kingsbury. 2021. On the limit of English conversational speech recognition. arXiv:2105.00982.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.00982 · arXiv 2105.00982

@misc{tuske2021limit,
  title = {{On the limit of English conversational speech recognition}},
  author = {Zoltan Tuske, George Saon and Brian Kingsbury},
  year = {2021},
  doi = {10.48550/arxiv.2105.00982},
  eprint = {2105.00982},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.00982}
}

References

55 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in ICLR, 2015. ↗
  2. H. K. J. Kuo et al., “End-to-end spoken language understanding without full transcripts,” in Interspeech, 2020, pp. 906–910. find
  3. A. Gulati et al., “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech, 2020, pp. 5036–5040.
  4. Z. Tüske et al., “Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard,” in Interspeech, 2020, pp. 551–555. find
  5. G. E. Hinton et al., “Improving neural networks by preventing co-adaptation of feature detectors,” 2012. [Online]. Available: http://arxiv.org/abs/1207.0580 ↗
  6. L. Wan et al., “Regularization of neural networks using DropConnect,” in ICML, vol. 28, no. 3, 2013, pp. 1058–1066. find
  7. N. Kanda, R. Takeda, and Y. Obuchi, “Elastic spectral distortion for low resource speech recognition with deep neural networks,” in ASRU, 2013, pp. 309–314. find
  8. T. Ko et al., “Audio augmentation for speech recognition,” in Interspeech, 2015, pp. 3586–3589. find
  9. G. Saon et al., “Sequence noise injected training for end-to-end speech recognition,” in ICASSP, 2019, pp. 6261–6265. find
  10. D. S. Park et al., “SpecAugment: A simple data augmentation method for automatic speech recognition,” in Interspeech, 2019, pp. 2613–2617.
  11. A. F. Murray and P. J. Edwards, “Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training,” IEEE Trans. on Neural Networks, vol. 5, no. 5, pp. 792– 802, 1994. ↗
  12. D. Krueger et al., “Zoneout: regularizing RNNs by randomly preserving hidden activations,” in ICLR, 2017. ↗
  13. C. Szegedy et al., “Rethinking the inception architecture for computer vision,” in CVPR, 2016, pp. 2818–2826. ↗
  14. A. Krogh and J. A. Hertz, “A simple weight decay can improve generalization,” in NIPS, 1992, pp. 950–957. find
  15. Y. Bengio et al., “Curriculum learning,” in ICML, 2009, pp. 41–48.
  16. S. Bengio et al., “Scheduled sampling for sequence prediction with recurrent neural networks,” in NIPS, 2015, pp. 1171–1179. ↗
  17. D. Amodei et al., “Deep speech 2: End-to-end speech recognition in English and Mandarin,” in ICML, 2016, pp. 173–182. ↗
  18. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015. ↗
  19. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR, 2019. find
  20. G. Saon et al., “Speaker adaptation of neural network acoustic models using i-vectors,” in ASRU, 2013, pp. 55–59. find
  21. M. Kitza et al., “Cumulative adaptation for BLSTM acoustic models,” in Interspeech, 2019, pp. 754–758. find
  22. G. Saon et al., “Advancing RNN transducer technology for speech recognition,” in ICASSP, 2021. find
  23. Z. Zhu, J. H. Engel, and A. Hannun, “Learning multiscale features directly from waveforms,” in Interspeech, 2016, pp. 1305–1309. find
  24. Z. Tüske, R. Schlüter, and H. Ney, “Acoustic modeling of speech waveform based on multi-resolution, neural network signal processing,” in ICASSP, 2018, pp. 4859–4863. find
  25. H. Hermansky, “Perceptual linear predictive (PLP) analysis of speech,” Journal of the Acoustical Society of America, vol. 87, no. 4, pp. 1738–1752, 1990. find
  26. R. Schlüter et al., “Gammatone features and feature combination for large vocabulary speech recognition,” in ICASSP, 2007, pp. 649–652. find
  27. C. Kim, K. Kim, and S. R. Indurthi, “Small energy masking for improved neural network training for end-to-end speech recognition,” in ICASSP, 2020, pp. 7684–7688. find
  28. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  29. M. Sperber et al., “Self-attentional acoustic models,” in Interspeech, 2018, pp. 3723–3727. find
  30. D. Povey et al., “A time-restricted self-attention layer for ASR,” in ICASSP, 2018, pp. 5874–5878. find
  31. Q. Zhang et al., “Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,” in Interspeech, 2020, pp. 7829–7833. ↗
  32. L. Dong, S. Xu, and B. Xu, “Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition,” in ICASSP, 2018, pp. 5884–5888. find
  33. R. Al-Rfou et al., “Character-level language modeling with deeper self-attention,” in AAAI Conf. on Artificial Intelligence, 2019, pp. 3159–3166. ↗
  34. Z. Dai et al., “Transformer-XL: Attentive language models beyond a fixed-length context,” in ACL, 2019, pp. 2978–2988. find
  35. Z. Tüske, R. Schlüter, and H. Ney, “Investigation on LSTM recurrent n-gram language models for speech recognition,” in Interspeech, 2018, pp. 3358–3362. ↗
  36. W. Xiong et al., “The Microsoft 2017 conversational speech recognition system,” in ICASSP, 2018, pp. 5934–5938. find
  37. K. Irie et al., “Training language models for long-span cross-sentence evaluation,” in ASRU, 2019, pp. 419–426. find
  38. E. McDermott, H. Sak, and E. Variani, “A density ratio approach to language model fusion in end-to-end automatic speech recognition,” in ASRU, 2019, pp. 434–441. find
  39. H. A. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach. Norwell, MA, USA: Kluwer Academic Publishers, 1993. find
  40. V. Manohar, D. Povey, and S. Khudanpur, “Semi-supervised maximum mutual information training of deep neural network acoustic models,” in Interspeech, 2015, pp. 2630–2634. find
  41. J. Kittler et al., “On combining classifiers,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226–239, 1998. ↗
  42. K. Kirchhoff and J. A. Bilmes, “Combination and joint training of acoustic classifiers for speech recognition,” in ASR2000, ISCA Tutorial and Research Workshop, 2000, pp. 17–23. find
  43. C. Dugast, L. Devillers, and X. Aubert, “Combining TDNN and HMM in a hybrid system for improved continuous-speech recognition,” IEEE Transactions on Speech and Audio Processing, vol. 2, no. 1, pp. 217–223, 1994. ↗
  44. Ç. Gülçehre et al., “On using monolingual corpora in neural machine translation,” 2015. [Online]. Available: http://arxiv.org/abs/1503.03535 ↗
  45. J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” 2016. [Online]. Available: http://arxiv.org/abs/1612.02695 ↗
  46. Z. Tu et al., “Modeling coverage for neural machine translation,” in ACL, 2016, pp. 76–85.
  47. A. Hannun et al., “Deep Speech: Scaling up end-to-end speech recognition,” 2014. [Online]. Available: http://arxiv.org/abs/1412.5567 ↗
  48. D. Povey et al., “The Kaldi speech recognition toolkit,” in ASRU, 2011. find
  49. G. Saon et al., “English conversational telephone speech recognition by humans and machines,” in Interspeech, 2017, pp. 132–136.
  50. A. Vaswani et al., “Attention is all you need,” in NIPS, 2017, pp. 5998–6008.
  51. J. K. Chorowski et al., “Attention-based models for speech recognition,” in NIPS, 2015, pp. 577–585. ↗
  52. W. Chan et al., “Listen, attend and spell: a neural network for large vocabulary conversational speech recognition,” in ICASSP, 2016, pp. 4960–4964. find
  53. P. Goyal et al., “Accurate, large minibatch SGD: Training imagenet in 1 hour,” 2017. [Online]. Available: http://arxiv.org/abs/1706.02677 ↗
  54. Y. Nesterov, “A method of solving a convex programming problem with convergence rate O(1/k 2 ),” in Soviet Mathematics Doklady, vol. 27, no. 2, 1983, pp. 372–376. find
  55. K. J. Han et al., “The CAPIO 2017 conversational speech recognition system,” 2018. [Online]. Available: http://arxiv.org/abs/1801.00059 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tuske-2021-limit-english-conversational-speech-recognition/