Elamur Research⌕
☰
home / papers / 2023 / VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System

VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System

Paper · 2023 · Abdul Waheed, Bashar Talafha, Peter Sullivan et al.

Summary

Arabic speech technology has mostly targeted Modern Standard Arabic (MSA). Systems built for a single dialect fail on speech from other regions. VoxArabica tackles this with a demo pipeline that first identifies the dialect of an utterance and then sends it to a matching speech recognizer.

The dialect identification (DID) component is HuBERT finetuned on ADI-17, plus only the MSA portions of ADI-5 and MGB-2. It chooses among 18 labels: 17 country-level dialects and MSA. For ASR, the authors finetuned Whisper (large-v2 and small) and XLS-R:

  • MSA models on Common Voice and MGB2,
  • an Egyptian model on MGB3 and FLEURS,
  • a Moroccan model on MGB5.

Dialects without a dedicated model fall back to zero-shot Whisper and MMS. A web interface supports file upload, recording, manual or automatic dialect selection, top-5 dialect confidences, and user flags on bad outputs.

The paper is a system demo. It shows example transcriptions but reports no WER or DID accuracy tables. Its value is a modular, openly usable architecture that brings dialect awareness to Arabic speech recognition.

Key points

  • DID covers 18 classes: MSA together with the dialects of Yemen, the UAE, Syria, Sudan, Qatar, Palestine, Oman, Morocco, Mauritania, Libya, Lebanon, Kuwait, Saudi Arabia, Jordan, Iraq, Egypt and Algeria.
  • The DID hyperparameters came from a 30-trial random search validated on the ADI-17 dev set. The best configuration was:
    • 192 freeze steps,
    • learning rate 6·10⁻⁴,
    • about 29k max steps,
    • 4.69 s audio crops,
    • thaw depth 3. LayerNorm/attention-only finetuning was tried and did not help.
  • ASR training setup:
    • AdamW, learning rate 1e-5, 500 warmup steps, no weight decay,
    • early stopping with patience 5,
    • DeepSpeed ZeRO stage-2 on 8×A100 GPUs. The MSA data was Common Voice 6.1, 9.0 and 11.0, together with MGB2.
  • Transcripts were normalized before training because the corpora were inconsistent (for example, some Quranic lines were fully diacritized and others not). Normalization:
    • removed punctuation except % and @,
    • removed diacritics, Hamzas and Maddas,
    • mapped Eastern Arabic numerals to Western ones,
    • dropped Latin script, since code-switching is out of scope.
  • Unlike FarSpeech, which supports only coarse DID and MSA-only ASR, VoxArabica is modular. Users can run DID alone, ASR alone, or both chained together.

Interesting ideas

  • I1. A classifier picks the variety and dispatches input to a specialist model; when no specialist exists, two generic models both run and their outputs are shown side by side. This gives graceful degradation instead of a silent wrong choice.
  • I2. When merging corpora with labels at different granularity, the authors kept only the part whose label maps cleanly (MSA from ADI-5) rather than guessing coarse-to-fine mappings, which avoids injecting label noise.
  • I3. During finetuning, the backbone stays frozen while the new head trains for a set number of steps, then only layers above a tuned "thaw depth" are unfrozen, and both values are searched as hyperparameters.
  • I4. Inconsistent annotation across sources was flattened by one deterministic normalization applied before training, so the model never learns arbitrary corpus-specific conventions.
  • I5. User-raised flags on wrong outputs act as a filter: flagged examples are discarded, and the rest become silver labels.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Detect the genre or English dialect of an input (e.g. tweets vs. legal text) and route it to a parser trained on matching treebanks; for uncovered cases, show the perceptron and gparse trees together.
  • I2 (learn): When building silver data from treebanks with differing annotation schemes, keep only the constructions whose conversion is unambiguous and skip the rest instead of forcing a mapping.
  • I3 (tree): When adapting gparse's transformer encoder to a new treebank, warm up the arc/label scorers on a frozen encoder first and tune how many lower layers stay frozen.
  • I4 (tokens): Apply one explicit, logged normalization pass (quotes, dashes, numerals, spelling variants) to all training corpora before tagging and lemmatization, so differences between corpora are not learned as linguistic signal.
  • I5 (learn): Let readers or tests flag wrong answers or trees, and exclude anything flagged from the silver set agreed by the witness parsers, as a cheap human negative filter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Abdul Waheed, Bashar Talafha, Peter Sullivan, AbdelRahim Elmadany and Muhammad Abdul-Mageed. 2023. VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.38

@inproceedings{waheed2023voxarabica,
  title = {{VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System}},
  author = {Abdul Waheed, Bashar Talafha, Peter Sullivan, AbdelRahim Elmadany and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.38},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.38}
}

References

46 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020a. Nadi 2020: The first nuanced arabic dialect identification shared task. arXiv preprint arXiv:2010.11334. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. Nadi 2021: The second nuanced arabic dialect identification shared task. arXiv preprint arXiv:2103.08466. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. Nadi 2022: The third nuanced arabic dialect identification shared task. arXiv preprint arXiv:2210.09582. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020b. Toward micro-dialect identification in diaglossic and code-switched environments. arXiv preprint arXiv:2010.04900. ↗
  6. Abdelrahman Ahmed, Yasser Hifny, Khaled Shaalan, and Sergio Toral. 2019. End-to-end lexicon free arabic speech recognition using recurrent neural networks. In Computational Linguistics, Speech And Image Processing For Arabic Language, pages 231– 248. World Scientific. ↗
  7. Abbas Raza Ali. 2020. Multi-dialect arabic speech recognition. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE.
  8. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279– 284. IEEE.
  9. Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, James Glass, Steve Renals, and Khalid Choukri. 2019. The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1026–1033. IEEE. find
  10. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic mgb-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316– 322. IEEE.
  11. Ahmed M. Ali, Hamdy Mubarak, and Stephan Vogel. 2014. Advances in dialectal arabic speech recognition: a study using twitter to improve egyptian asr. In International Workshop on Spoken Language Translation.
  12. Hamzah A Alsayadi, Abdelaziz A Abdelhamid, Islam Hegazy, Bandar Alotaibi, and Zaki T Fayed. 2022. Deep investigation of the recent advances in dialectal arabic speech recognition. IEEE Access, 10:57063– 57079. ↗
  13. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2022. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeech 2022, pages 2278–2282.
  14. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296. ↗
  15. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  16. Yonatan Belinkov, Ahmed Ali, and James Glass. 2019. Analyzing phonetic and graphemic representations in end-to-end automatic speech recognition. arXiv preprint arXiv:1907.04224. ↗
  17. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  18. Weicheng Cai, Zexin Cai, Wenbo Liu, Xiaoqi Wang, and Ming Li. 2018. Insights in-to-end learning scheme for language identification. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5209–5213. IEEE. find
  19. Shammur A Chowdhury, Ahmed Ali, Suwon Shon, and James R Glass. 2020. What does an end-to-end dialect identification model learn about non-dialectal information? In INTERSPEECH, pages 462–466. find
  20. Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021. Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR. In Proc. Interspeech 2021, pages 2466–2470.
  21. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE. find
  22. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE.
  23. Shervin Malmasi and Marcos Zampieri. 2017. Arabic dialect identification using iVectors and ASR transcripts. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 178–183, Valencia, Spain. Association for Computational Linguistics.
  24. Najim Dehak, Patrick J Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet. 2010. Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing, 19(4):788–798. ↗
  25. Raphael Olivier and Bhiksha Raj. 2022. There is more than one kind of robustness: Fooling whisper with adversarial examples. arXiv preprint arXiv:2210.17316. ↗
  26. Amira Dhouib, Achraf Othman, Oussama El Ghoul, Mohamed Koutheair Khribi, and Aisha Al Sinani. 2022. Arabic automatic speech recognition: A systematic literature review. Applied Sciences, 12(17). ↗
  27. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  28. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  29. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516. ↗
  30. Mohamed Eldesouki, Naassih Gopee, Ahmed Ali, and Kareem Darwish. 2019. Farspeech: Arabic natural language processing for live arabic speech. In IN-TERSPEECH, pages 2372–2373. find
  31. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. ↗
  32. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461.
  33. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2019. Zero: Memory optimization towards training A trillion parameter models. CoRR, abs/1910.02054. ↗
  34. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. ↗
  35. Amir Hussein, Shinji Watanabe, and Ahmed Ali. 2022. Arabic speech recognition by end-to-end, modular systems and human. Computer Speech & Language, 71:101272. ↗
  36. Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2020. Multilingual speech translation with efficient finetuning of pretrained models. arXiv preprint arXiv:2010.12829. ↗
  37. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  38. Khaled Lounnas, Hassan Satori, Mohamed Hamidi, Hocine Teffahi, Mourad Abbas, and Mohamed Lichouri. 2020. Cliasr: a combined automatic speech recognition and language identification system. In 2020 1st international conference on innovative research in applied science, engineering and Technology (IRASET), pages 1–5. IEEE. find
  39. Anthony Rousseau, Paul Deléglise, and Yannick Esteve. 2012. Ted-lium: an automatic speech recognition dedicated corpus. In LREC, pages 125–129. find
  40. Suwon Shon, Ahmed Ali, Younes Samih, Hamdy Mubarak, and James Glass. 2020. Adi17: A fine-grained arabic dialect identification dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8244–8248. IEEE. find
  41. David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018. X-vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5329–5333. IEEE. find
  42. Peter Sullivan, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2023. On the Robustness of Arabic Speech Dialect Identification. In Proc. IN-TERSPEECH 2023, pages 5326–5330.
  43. Bashar Talafha, Abdul Waheed, and Muhammad Abdul-Mageed. 2023. N-shot benchmarking of whisper on diverse arabic speech recognition. arXiv preprint arXiv:2306.02902. ↗
  44. Dong Wang, Xiaodong Wang, and Shaohe Lv. 2019. An overview of end-to-end automatic speech recognition. Symmetry, 11(8):1018. ↗
  45. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  46. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath, Pedro Moreno, Chung-Cheng Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. 2023. Google usm: Scaling automatic speech recognition beyond 100 languages. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/waheed-2023-voxarabica-robust-dialect-aware-arabic/