Elamur Research⌕
☰
home / papers / 2018 / Attention-based sequence-to-sequence model for speech recognition: development of state-of-the-art system on LibriSpeech and its application to non-native English

Attention-based sequence-to-sequence model for speech recognition: development of state-of-the-art system on LibriSpeech and its application to non-native English

Paper · 2018 · Yan Yin, Ramon Prieto, Bin Wang et al.

Summary

The authors ask whether an end-to-end attention model can match or beat classic hybrid HMM/NN recognizers, especially for English spoken by learners. Learners' pronunciation varies widely, and a fixed lexicon handles that poorly. They build a Listen, Attend and Spell (LAS) system: a pyramid BLSTM encoder that shortens the time axis, and an attention-based LSTM decoder that outputs word pieces. They then add a series of training improvements:

  • location-aware attention
  • scheduled sampling
  • label smoothing
  • careful learning-rate and gradient control
  • BPE word pieces
  • MWER discriminative training
  • speed-perturbation augmentation
  • external LMs, used through shallow fusion or n-best rescoring

On LibriSpeech the final system reaches 3.43% WER on test-clean, which was a new best at the time. On in-house Liulishuo data from Chinese speakers of English, LAS roughly equals a recent Kaldi recipe. It is slightly behind on read speech and ahead on spontaneous speech. This suggests that learning pronunciation jointly with acoustics is promising for non-native ASR.

Key points

  • Architecture: 40-dim filter-bank input, 3 pyramid BLSTM layers with 1024 units per direction, a 2-layer LSTM decoder with 512 units, roughly 500 BPE word pieces, and TensorFlow training on 4 GPUs.
  • The LibriSpeech ablation without an external LM lowers test-clean WER from 6.61% (character baseline) to 4.34%. Stabilized training alone contributes about 13% relative reduction, and location-aware attention about 9.2%.
  • Gradient control combines two clipping rules:
    • a moving-average norm tracker that clips norms more than 2 standard deviations above the mean;
    • a fixed norm cap of 5.0, which helps especially with multi-GPU stability.
  • MWER training starts after CE converges. It is interpolated with CE and computed over n-best lists, with scores flattened by an exponent below one. The 90%/110% speed-perturbed copies are used only in this stage.
  • Liulishuo results:
SystemRead WERSpontaneous WER
LAS, no external LM10.44%15.16%
LAS + LSTM LM10.41%14.77%
Kaldi + trigram11.47%16.47%
Kaldi + LSTM LM9.95%15.44%

External LMs helped LAS little, probably because their mostly news-domain text does not match the test speech.

Interesting ideas

  • I1. Report a cumulative ablation ladder, where each technique is stacked on the previous one and its relative error reduction is shown, so the contribution of every trick is visible rather than buried in one final number.
  • I2. Clip gradients adaptively against a running estimate of the norm distribution, and keep a hard static cap as a backstop for spikes the tracker misses.
  • I3. Fine-tune with an expected-task-metric loss over n-best outputs, flatten the dominant top hypothesis with an exponent below one, and mix in the ordinary CE loss for stability.
  • I4. Delay expensive or mismatch-reducing tricks until the cheap base training has converged: scheduled sampling starts once validation loss plateaus, and augmented data enters only in the final stage.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluations of parser or reader changes could follow the same incremental table, with relative gains, so we can tell which repair rules or training tweaks actually pay off.
  • I2 (tree): Training of the gparse transformer parser, particularly on multiple GPUs, could use this dual clipping to avoid occasional divergence without tuning one fixed threshold.
  • I3 (tree): gparse could be fine-tuned on n-best trees to minimize expected labeled attachment error directly, which is closer to how we score parses than token-level cross-entropy.
  • I4 (learn): Silver data and LLM-teacher tiers could be staged the same way, training first on clean gold treebanks and adding noisier or larger material only during late fine-tuning to save compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSpeech & spoken language · 233 works

Cite

Cite as Yan Yin, Ramon Prieto, Bin Wang, Jianwei Zhou, Yiwei Gu, Yang Liu and Hui Lin. 2018. Attention-based sequence-to-sequence model for speech recognition: development of state-of-the-art system on LibriSpeech and its application to non-native English. arXiv:1810.13088.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1810.13088 · arXiv 1810.13088

@misc{yin2018attention,
  title = {{Attention-based sequence-to-sequence model for speech recognition: development of state-of-the-art system on LibriSpeech and its application to non-native English}},
  author = {Yan Yin, Ramon Prieto, Bin Wang, Jianwei Zhou, Yiwei Gu, Yang Liu and Hui Lin},
  year = {2018},
  doi = {10.48550/arxiv.1810.13088},
  eprint = {1810.13088},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.13088}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent Pretrained Deep Neural Networks for large vocabulary speech recognition,” vol. 20, January 2012, pp. 30–42. find
  2. O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolutional Neural Networks for Speech Recognition,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 22, pp. 1533–1545, October 2014. ↗
  3. T. N. Sainath, O. Vinyals, A. W. Senior, and H. Sak, “Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks,” in ICASSP 2015, 2015, pp. 4580– 4584. find
  4. D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y.Wang, and S. Khudanpur, “Purely Sequence-Trained Neural Networks for ASR based on Lattice-free MMI,” in Interspeech 2016, 2016, pp. 2751– 2755. find
  5. A. Graves, “Sequence Transduction with Recurrent Neural Networks,” CoRR, vol. abs/1211.3711, 2012. ↗
  6. W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, Attend and Spell,” CoRR, vol. abs/1508.01211, 2015. ↗
  7. N. Jaitly, D. Sussillo, Q. V. Le, O. Vinyals, I. Sutskever, and S. Bengio, “An Online Sequence-to-Sequence Model Using Partial Conditioning,” in Proc. NIPS, 2016. find
  8. H. Sak, M. Shannon, K. Rao, and F. Beaufays, “Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping,” in Proc. Interspeech, 2017. find
  9. C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, K. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, “State-ofthe-art Speech Recognition with Sequence-to-Sequence Models,” CoRR, abs/1712.01769, 2017. ↗
  10. C. Weng, J. Cui, G. Wang, J. Wang, C. Yu, D. Su, D. Yu, “Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition,” in INTERSPEECH, 2018. find
  11. A. Zeyer, K. Irie, R. Schluter, and H. Ney, “Improved Training of End-to-End Attention Models for Speech Recognition,” arXiv:1805.03294, 2018. ↗
  12. R. Prabhavalkar, K. Rao, T. N. Sainath, B. Li, L. Johnson, and N. Jaitly, “A Comparison of Sequence-to-sequence Models for Speech Recognition,” in Proc. Interspeech, 2017. find
  13. J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-Based Models for Speech Recognition,” arXiv:1506.07503, 2015. ↗
  14. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” CoRR, vol. abs/1506.03099, 2015. ↗
  15. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. ↗
  16. R. Sennrich, B. Haddow, and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” in ACL, Berlin, Germany, August 2016, pp. 1715–1725.
  17. K. Vesely, A. Ghoshal, L. Burget, and D. Povey. Sequence-Discriminative Training of Deep Neural Networks. In Proceedings of Interspeech . 2013. ↗
  18. B. Kingsbury, “Lattice-Based Optimization of Sequence Classification Criteria for Neural-Network Acoustic Modeling,” in Proc. ICASSP, 2009. find
  19. C. Gulcehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On Using Monolingual Corpora in Neural Machine Translation,” CoRR, vol. abs/1503.03535, 2015. ↗
  20. M. Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” Available online:http://download.tensorflow.org/paper/whitepaper201 5.pdf, 2015. ↗
  21. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: an ASR Corpus based on Public Domain Audio Books,” in ICASSP. IEEE, 2015, pp.5206–5210. find
  22. Wu et. al., “Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. ↗
  23. A. Rousseau, P. Deléglise, and Y. Estève, "TED-LIUM: an automatic speech recognition dedicated corpus", in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), 2012. find
  24. E. Grave, A. Joulin, M Cissé, D. Grangier, H. Jégou, “Efficient softmax approximation for GPUs”. arXiv preprint arXiv:1609.04309. 2016. ↗
  25. D. P. Kingma, J. Ba, “Adam: A method for stochastic optimization”, arXiv preprint arXiv:1412.6980. 2014. ↗
  26. X. Yang, J. Li, and X. Zhou, “A Novel Pyramidal-FSMN Architecture with Lattice-free MMI for Speech Recognition”, arXiv preprint https://arxiv.org/abs/1810.11352, 2018. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yin-2018-attention-based-sequence-sequence-model/