Elamur Research⌕
☰
home / papers / 2018 / Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition

Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition

Paper · 2018 · Xuesong Yang, Kartik Audhkhasi, Andrew Rosenberg et al.

Summary

Speech recognizers lose accuracy when test speakers have an accent that differs from the training data. Earlier multi-accent acoustic models relied on accent information only indirectly, for example through separate phone sets per accent. Yang et al. instead train an accent identification (AID) network together with a multi-task acoustic model. Both are built on bidirectional LSTMs. The acoustic part uses CTC loss, with shared lower layers and a separate DNN output branch for each accent. The AID part pools frame representations into one utterance-level decision. In the joint model, AID serves as an auxiliary objective on the lower layers, and the two losses are mixed with a tuned weight. At test time, the AID output acts as a hard switch that picks which accent-specific output layer to use. On the WSJ American English and Cambridge British English corpora, the joint model beats a strong multi-task baseline, with relative WER gains of 5.94% on British and 9.47% on American English. Accent identification also improves under joint training. The work suggests that explicit accent supervision helps acoustic modeling.

Key points

  • Data: WSJ-1 (American English) and Cambridge (British English), about 15 hours each. Their phone inventories overlap but differ, with 42 and 45 phones. 5% of the training data was held out for development.
  • Features: 40-dimensional log-Mel filter banks with per-utterance cepstral mean subtraction and no VTLN. Neighbouring frames are stacked into 80-dimensional vectors at half the frame rate.
  • The AID model uses BLSTMs, DNN projections and average pooling, with a single sigmoid output for US vs UK. It is trained with cross-entropy and thresholded at 0.5 to select the output head.
  • Joint objective: (1−α)·L_AM (CTC) + α·L_AID (cross-entropy). Because the two losses sit on different scales, α is tuned on development data.
  • Results: relative WER reductions of 5.94% (UK) and 9.47% (US) over the multi-task AM. The switch was tested with both oracle and predicted accent labels.

Interesting ideas

  • I1. Supervise the lower layers with an auxiliary "which variety is this?" task while the main task sits higher, so shared layers explicitly encode variety rather than discovering it on their own.
  • I2. Keep one shared trunk with separate variety-specific output heads, and let a visible classifier decision choose the head for each input. The choice is a discrete, inspectable step rather than a hidden blend.
  • I3. When the evidence for a property is not present at every position, pool over the whole sequence before classifying instead of deciding locally.
  • I4. When two losses on different scales are combined, treat the mixing weight as a tuned hyperparameter selected on held-out data rather than fixing it at 0.5.
  • I5. Evaluate the pipeline with both the gold (oracle) upstream decision and the predicted one, so the cost of upstream errors is measured separately.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could add a treebank/genre-source classifier on its lower encoder layers, which would help it separate UD dialect conventions before predicting arcs.
  • I2 (tree): treebank-specific arc/label heads could be selected by a predicted source label, and that label logged so every parse states which convention it followed.
  • I3 (tokens): register or variety detection (e.g., British vs American spelling, informal text) could pool over all tokens of a sentence or document before choosing lemmatization or spelling rules.
  • I4 (learn): multi-objective training of the parser (arcs, labels, auxiliary tags) could tune its weights on a dev split alongside the negative-control evaluation.
  • I5 (learn): run frames and answer evaluations once on gold UD trees and once on predicted trees, to attribute failures to the correct stage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations🏢IBMArmonk · United States85 works · 66,118 citations🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 233 works

Cite

Cite as Xuesong Yang, Kartik Audhkhasi, Andrew Rosenberg, Samuel Thomas, Bhuvana Ramabhadran and Mark Hasegawa-Johnson. 2018. Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition. arXiv:1802.02656.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.02656 · arXiv 1802.02656

@misc{yang2018joint,
  title = {{Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition}},
  author = {Xuesong Yang, Kartik Audhkhasi, Andrew Rosenberg, Samuel Thomas, Bhuvana Ramabhadran and Mark Hasegawa-Johnson},
  year = {2018},
  doi = {10.48550/arxiv.1802.02656},
  eprint = {1802.02656},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.02656}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig, “Achieving human parity in conversational speech recognition,” arXiv preprint arXiv:1610.05256, 2016. ↗
  2. George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, et al., “English conversational telephone speech recognition by humans and machines,” arXiv preprint arXiv:1703.02136, 2017. ↗
  3. Chao Huang, Tao Chen, and Eric Chang, “Accent issues in large vocabulary continuous speech recognition,” International Journal of Speech Technology, vol. 7, no. 2, pp. 141–153, 2004. ↗
  4. Janet Holmes, An introduction to sociolinguistics, Routledge, 2013. find
  5. Victor Soto, Olivier Siohan, Mohamed Elfeky, and Pedro Moreno, “Selection and combination of hypotheses for dialectal speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 5845–5849. find
  6. Fadi Biadsy and Julia Hirschberg, “Using prosody and phonotactics in arabic dialect identification,” in Tenth Annual Conference of the International Speech Communication Association, 2009. find
  7. Mohamed Elfeky, Meysam Bastani, Xavier Velez, Pedro Moreno, and Austin Waters, “Towards acoustic model unification across dialects,” in Spoken Language Technology Workshop (SLT), 2016 IEEE. IEEE, 2016, pp. 624–628. find
  8. Kanishka Rao and Haşim Sak, “Multi-accent speech recognition with hierarchical grapheme based models,” in Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 2017, pp. 4815–4819. find
  9. Yan Huang, Dong Yu, Chaojun Liu, and Yifan Gong, “Multi-accent deep neural network acoustic model with accent-specific top layer using the kld-regularized model adaptation,” in Fifteenth Annual Conference of the International Speech Communication Association, 2014. find
  10. Mingming Chen, Zhanlei Yang, Jizhong Liang, Yanpeng Li, and Wenju Liu, “Improving deep neural networks based multi-accent mandarin speech recognition using i-vectors and accent-specific top layer,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  11. Jiangyan Yi, Hao Ni, Zhengqi Wen, Bin Liu, and Jianhua Tao, “Ctc regularized model adaptation for improving lstm rnn based multi-accent mandarin speech recognition,” in Chinese Spoken Language Processing (ISCSLP), 2016 10th International Symposium on. IEEE, 2016, pp. 1–5. ↗
  12. Yanli Zheng, Richard Sproat, Liang Gu, Izhak Shafran, Haolang Zhou, Yi Su, Daniel Jurafsky, Rebecca Starr, and Su-Youn Yoon, “Accent detection and speech recognition for shanghai-accented mandarin.,” in Interspeech, 2005, pp. 217– 220. find
  13. Mohamed Elfeky, Pedro Moreno, and Victor Soto, “Multi-dialectical languages effect on speech recognition: Too much choice can hurt,” in International Conference on Natural Language and Speech Processing (ICNLSP), 2015. ↗
  14. Janet Pierrehumbert, “Phonological representation: Beyond abstract versus episodic,” Annu. Rev. Linguist., vol. 2, pp. 33– 52, 2016. ↗
  15. Alex Graves and Navdeep Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014, pp. 1764–1772. find
  16. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al., “Deep speech 2: End-to-end speech recognition in english and mandarin,” in International Conference on Machine Learning, 2016, pp. 173–182. ↗
  17. Haşim Sak, Andrew Senior, Kanishka Rao, and Françoise Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” in Proceedings of INTER-SPEECH, 2015. find
  18. Yajie Miao, Mohammad Gowayyed, and Florian Metze, “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,” in Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on. IEEE, 2015, pp. 167–174. find
  19. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al., “Deep speech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.5567, 2014. ↗
  20. Andrew Maas, Ziang Xie, Dan Jurafsky, and Andrew Ng, “Lexicon-free conversational speech recognition with neural networks,” in Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015, pp. 345–354. find
  21. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “End-to-end continuous speech recognition using attention-based recurrent nn: first results,” arXiv preprint arXiv:1412.1602, 2014. ↗
  22. Liang Lu, Xingxing Zhang, Kyunghyun Cho, and Steve Renals, “A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition.,” in INTERSPEECH, 2015, pp. 3249–3253. find
  23. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 4945–4949. find
  24. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 4960–4964. find
  25. K. Audhkhasi, B. Ramabhadran, G. Saon, M. Picheny, and D. Nahamoo, “Direct acoustics-to-word models for English conversational speech recognition,” in Proc. Interspeech, 2017, pp. 959–963.
  26. H. Soltau, G. Saon, and B. Kingsbury, “The ibm attila speech recognition toolkit,” in Spoken Language Technology Workshop (SLT), 2010, pp. 97–102. find
  27. Diederik Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. ↗
  28. H. Bourlard and N. Morgan, “Generalization and parameter estimation in feedforward nets: Some experiments,” in Advances in Neural Information Processing Systems, 1990, vol. II, pp. 630–637. find
  29. Hal Daumé III, “Frustratingly easy domain adaptation,” arXiv preprint arXiv:0907.1815, 2009. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2018-joint-modeling-accents-acoustics-multi/