Elamur Research⌕
☰
home / papers / 2021 / Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR

Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR

Summary

The paper asks whether one end-to-end speech recognizer, trained on several languages at once, can cope with how Arabic is actually spoken. Real speech mixes Modern Standard Arabic (MSA) with regional dialects and often switches into English or French mid-sentence. The authors trained a conformer encoder with transformer decoders on roughly 1,000 hours of audio:

  • Arabic broadcast news from QASR,
  • English from TEDLIUM3, LibriSpeech and MGB-1,
  • French from CommonVoice.

They did not supply any language or dialect identity signal. They evaluated the model on monolingual sets, on dialectal sets (Egyptian MGB3, Moroccan MGB5) and on code-switching sets, both cross-lingual and MSA–Egyptian. They also compared input symbol designs: characters vs word-pieces, and shared vs per-language Latin symbols. Because one word may appear in different scripts, they added a transliteration-aware error metric. The model beats prior state-of-the-art Arabic end-to-end systems on dialectal and code-switching speech, even though its training data contained almost no switching. Two new code-switching test sets are released.

Key points

  • The architecture is a conformer with 12 encoder and 6 decoder layers, 8 attention heads, 512-dimensional attention and a 10K BPE vocabulary. It is trained with a joint CTC + cross-entropy objective, with α = 0.3.
  • Training uses about 996.6 hours in total. This includes a 642-hour QASR subset (about 70% MSA, the rest dialects), 200 hours of English and 100 hours of French.
  • The new code-switching test sets are ESCWA.CS (about 2.8 h of UN meeting speech, Arabic–English/French, corpus-level CMI 28) and QASR.CS (about 5.9 h of broadcast news, CMI 30.5).
  • Dialectal code-switching is tested on DACS (about 1.5 h, MSA↔Egyptian, CMI 36.5).
  • Results are reported both as standard WER and as transliterated WER (TW). TW does not penalize a correct word written in the "other" script.

Interesting ideas

  • I1. The authors score output with a transliteration-aware WER so that a correct word is not counted as wrong just because of script choice; this separates real recognition errors from arbitrary surface variation. — For Mova (learn): Evaluation could normalize harmless surface variants before scoring (digits vs number words, British vs American spelling), so that measured regressions reflect real errors.
  • I2. The Code-Mixing Index measures, at corpus level, how much a dataset mixes languages; this turns a vague property into a number that can be compared across test sets. — For Mova (tree): A similar index could quantify how much a treebank or silver corpus mixes genres or UD annotation dialects, which would explain parser score gaps between sets.
  • I3. One model trained jointly on related varieties handles mixed input without any explicit language-ID flag, so no brittle routing step is needed. — For Mova (tokens): The tagger and morphology could be trained on pooled registers (formal, informal, archaic tales) without a genre switch, testing whether one model beats per-register routing.
  • I4. The paper runs a controlled comparison of shared vs distinct symbol inventories across languages; this shows whether sharing helps transfer or causes confusion. — For Mova (tree): When converting between UD dialects, run the same experiment with shared vs separate label sets in gparse to decide where labels should merge.
  • I5. A joint CTC + attention loss is used as a multi-task objective to make the model more robust. — For Mova (tree): gparse could add an auxiliary per-token objective (e.g., head-distance or tag prediction) alongside the main parse loss to stabilize training on noisy silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

28 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.14779 · arXiv 2105.14779

@misc{chowdhury2021one,
  title = {{Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR}},
  author = {Chowdhury, Shammur Absar and Hussein, Amir and Abdelali, Ahmed and Ali, Ahmed},
  year = {2021},
  doi = {10.48550/arxiv.2105.14779},
  eprint = {2105.14779},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.14779}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Datta, B. Ramabhadran, J. Emond, A. Kannan, and B. Roark, “Language-agnostic multilingual modeling,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 8239–8243. find
  2. V. Pratap, A. Sriram, P. Tomasello, A. Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert, “Massively multilingual asr: 50 languages, 1 model, 1 billion parameters,” arXiv preprint arXiv:2007.03001, 2020. ↗
  3. B. Li, Y. Zhang, T. Sainath, Y. Wu, and W. Chan, “Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5621–5625. find
  4. A. Kannan, A. Datta, T. N. Sainath, E. Weinstein, B. Ramabhadran, Y. Wu, A. Bapna, Z. Chen, and S. Lee, “Large-scale multilingual speech recognition with a streaming end-to-end model,” Proc. Interspeech 2019, pp. 2130–2134, 2019. find
  5. S. Toshniwal, T. N. Sainath, R. J. Weiss, B. Li, P. Moreno, E. Weinstein, and K. Rao, “Multilingual speech recognition with a single end-to-end model,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4904–4908. find
  6. J. Cho, M. K. Baskar, R. Li, M. Wiesner, S. H. Mallidi, N. Yalta, M. Karafiat, S. Watanabe, and T. Hori, “Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling,” in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 521–527. find
  7. H. Bourlard, J. Dines, M. Magimai-Doss, P. N. Garner, D. Imseng, P. Motlicek, H. Liang, L. Saheer, and F. Valente, “Current trends in multilingual speech processing,” Sadhana, vol. 36, no. 5, pp. 885–915, 2011. find
  8. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, “Multilingual acoustic models using distributed deep neural networks,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 8619–8623. find
  9. H. Lin, L. Deng, D. Yu, Y.-f. Gong, A. Acero, and C.-H. Lee, “A study on multilingual acoustic modeling for large vocabulary asr,” in 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2009, pp. 4333–4336. find
  10. L. Burget, P. Schwarz, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. Goel, M. Karafiát, D. Povey et al., “Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models,” in 2010 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2010, pp. 4334–4337. find
  11. A. Graves, “Sequence transduction with recurrent neural networks,” arXiv preprint arXiv:1211.3711, 2012. ↗ our card
  12. Y. Li and P. Fung, “Improved mixed language speech recognition using asymmetric acoustic model and language model with code-switch inversion constraints,” in ICASSP, 2013. find
  13. G. Sreeram and R. Sinha, “Exploration of end-to-end framework for code-switching speech recognition task: Challenges and enhancements,” IEEE Access, vol. 8, pp. 68 146–68 157, 2020. find
  14. D. Amazouz, M. Adda-Decker, and L. Lamel, “Addressing code-switching in French/Algerian Arabic speech,” in Interspeech, 2017. find
  15. W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 4960– 4964. find
  16. X. Zhou, E. Yılmaz, Y. Long, Y. Li, and H. Li, “Multi-encoder-decoder transformer for code-switching speech recognition,” arXiv preprint arXiv:2006.10414, 2020. ↗
  17. E. S. Badawi, M. Carter, and A. Gully, Modern written Arabic: A comprehensive grammar. Routledge, 2013. find
  18. A. Ali, S. Chowdhury, M. Afify, W. El-Hajj, H. Hajj, M. Abbas, A. Hussein, N. Ghneim, M. Abushariah, and A. Alqudah, “Connecting arabs: bridging the gap in dialectal speech recognition,” Communications of the ACM, vol. 64, no. 4, pp. 124–129, 2021. find
  19. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. ↗ our card
  20. P. Ghahremani, B. BabaAli, D. Povey, K. Riedhammer, J. Trmal, and S. Khudanpur, “A pitch extraction algorithm tuned for automatic speech recognition,” in IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2014, pp. 2494–2498. find
  21. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006. find
  22. P. Bell, M. J. Gales, T. Hain, J. Kilgour, P. Lanchantin, X. Liu, A. McParland, S. Renals, O. Saz, M. Wester et al., “The mgb challenge: Evaluating multi-genre broadcast media recognition,” in 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015, pp. 687–693. find
  23. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019. ↗ our card
  24. H. Mubarak, A. Hussein, S. A. Chowdhury, and A. Ali, “Qasr: Qcri aljazeera speech resource a large scale annotated arabic speech corpus,” in ACL, 2021. find
  25. S. A. Chowdhury, Y. Samih, M. Eldesouki, and A. Ali, “Effects of dialectal code-switching on speech modules: A study using egyptian Arabic broadcast speech,” Proc. Interspeech, 2020. find
  26. B. Gambäck and A. Das, “Comparing the Level of Code-Switching in Corpora,” in LREC, 2016. find
  27. A. Ali, S. Vogel, and S. Renals, “Speech recognition challenge in the wild: Arabic mgb-3,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 316–322. find our card
  28. A. Ali, S. Shon, Y. Samih, H. Mubarak, A. Abdelali, J. Glass, S. Renals, and K. Choukri, “The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 1026–1033. find
  29. T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  30. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019. ↗ our card
  31. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, . Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, 2017, pp. 5998–6008. find
  32. T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” arXiv preprint arXiv:1808.06226, 2018. ↗ our card
  33. A. Hussein, S. Watanabe, and A. Ali, “Arabic speech recognition by end-to-end, modular systems and human,” arXiv preprint arXiv:2101.08454, 2021. ↗ our card
  34. F. Dalvi, Y. Zhang, S. Khurana, N. Durrani, H. Sajjad, A. Abdelali, H. Mubarak, A. Ali, and S. Vogel, “QCRI live speech translation system,” in Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain: Association for Computational Linguistics, Apr. 2017, pp. 61–64. [Online]. Available: https://www.aclweb.org/anthology/E17-3016 ↗
  35. A. Ali, S. A. Chowdhury, A. Hussein, and Y. Hifny, “Arabic code-switching speech recognition using monolingual data,” Proc. Interspeech 2021, 2021. find our card
  36. Y. Samih, M. Attia, M. Eldesouki, A. Abdelali, H. Mubarak, L. Kallmeyer, and K. Darwish, “A neural architecture for dialectal Arabic segmentation,” in Proceedings of the Third Arabic Natural Language Processing Workshop. Valencia, Spain: Association for Computational Linguistics, Apr. 2017, pp. 46–54. [Online]. Available: https://www.aclweb.org/anthology/W17-1306 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2021-towards-one-model-rule-all/