Elamur Research⌕
☰
home / papers / 2024 / Casablanca: Data and Models for Multidialectal Arabic Speech Recognition

Casablanca: Data and Models for Multidialectal Arabic Speech Recognition

Paper · 2024 · Bashar Talafha, Karima Kadaoui, Samar Mohamed Magdy et al.

Summary

Arabic speech recognition still lacks reliable, manually labeled data for regional varieties. Most large Arabic corpora are dominated by MSA or were transcribed with light supervision. Casablanca addresses this with a community-built corpus of about 48 hours of speech from YouTube TV-series episodes. Its eight country-level varieties span North Africa, the Levant and the Gulf: Moroccan, Algerian, Mauritanian, Egyptian, Palestinian, Jordanian, Emirati and Yemeni. Several of these, notably Emirati, Yemeni and Mauritanian, had little or no prior representation. Native-speaker annotators transcribed each snippet by hand and tagged it with speaker gender, dialect and code-switching. Code-switched English and French words appear in both Latin and Arabic script. The authors benchmark Whisper (v2 and v3), SeamlessM4T, MMS and Arabic-adapted Whisper variants, in both zero-shot and Arabic-finetuned settings. They also run zero-shot dialect identification and gender recognition. Results show that current systems struggle with dialectal and code-switched speech. Whisper never reproduces Latin-script switches, and the dialect classifier transfers poorly to the drama domain. The resource gives the field a fine-grained, fully supervised benchmark for dialectal Arabic.

Key points

  • 48 hours of fully human-transcribed speech across 8 dialects. The authors present it as the largest fully supervised Arabic dialect speech dataset, larger than MGB-3 and MGB-5.
  • Data pipeline:
    • 15 native speakers chose TV episodes longer than 15 minutes.
    • pyannote VAD cut the audio into snippets.
    • 27 annotators with NLP graduate backgrounds labeled the snippets in LabelStudio, with at least two annotators per dialect from different regions.
  • Only YouTube URLs, timestamps and annotations are released, because of copyright. Every segment carries a gender label, and code-switching (EN+FR) is given with transliteration.
  • On code-switched segments, whisper-lg-v3 scores about 90.89 WER with automatic language detection. Forcing English raises WER to about 131.54, because the model translates instead of transcribing.
  • Zero-shot dialect identification with a HuBERT model reaches 36.44% accuracy. An XLS-R gender classifier trained on Librispeech reaches 83.56%. Male speakers make up over 60% in every dialect except Moroccan.

Interesting ideas

  • I1. Each code-switched word is stored twice, once in its original Latin form and once transliterated into Arabic script, so outputs can be scored against either convention.
  • I2. Setting the decoding language to English made Whisper translate silently instead of transcribing, which only came to light by checking outputs condition by condition.
  • I3. Each dialect had at least two annotators from different regions of the same country, to capture sub-dialect variation in spelling and vocabulary.
  • I4. Before transcription, every snippet was triaged as Dialect, MSA or Other, and only clear segments were kept.
  • I5. The corpus is distributed as source pointers plus timestamps rather than the media itself, which sidesteps copyright while keeping annotations reproducible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Foreign or borrowed tokens could keep both a surface form and a normalized form, so lemmatization and evaluation stay fair whichever spelling a text uses.
  • I2 (learn): Add negative-control checks that confirm a component performed the requested operation, rather than a plausible neighbouring task that still scores somewhat.
  • I3 (tree): When building silver treebanks or UD dialect conversions, draw witnesses from deliberately different "regions" (genres, parser families) so their agreement reflects real robustness.
  • I4 (tokens): A cheap register or noise triage step before tagging could route non-standard or garbled sentences away from the main parser and keep downstream errors explainable.
  • I5 (answer): QA sets built on copyrighted tales could ship as offsets into source editions plus Mova's annotations, allowing the evaluation data to be shared.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations🎓Ain Shams UniversityCairo · Egypt12 works · 109 citations🎓Institut National de Statistique et d'Economie AppliquéeRabat · Morocco4 works · 64 citations🎓Université de NouakchottNouakchott · Mauritania2 works · 82 citations

Also on the byline: Technische Hochschule Mittelhessen.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Bashar Talafha, Karima Kadaoui, Samar Mohamed Magdy, Mariem Habiboullah, Chafei Mohamed Chafei, Ahmed Oumar El-Shangiti, Hiba Zayed, Mohamedou Cheikh Tourad, Rahaf Alhamouri, Rwaa Assi, Aisha Alraeesi, Hour Mohamed, Fakhraddin Alwajih, Abdelrahman Mohamed, Abdellah El Mekki, El Moatez Billah Nagoudi, Benelhadj Djelloul Mama Saadia, Hamzah A. Alsayadi, Walid Al-Dhabyani, Sara Shatnawi, Yasir Ech-chammakhy, Amal Makouar, Yousra Berrachedi, Mustafa Jarrar, Shady Shehata, Ismail Berrada and Muhammad Abdul-Mageed. 2024. Casablanca: Data and Models for Multidialectal Arabic Speech Recognition. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.1211

@inproceedings{talafha2024casablanca,
  title = {{Casablanca: Data and Models for Multidialectal Arabic Speech Recognition}},
  author = {Bashar Talafha, Karima Kadaoui, Samar Mohamed Magdy, Mariem Habiboullah, Chafei Mohamed Chafei, Ahmed Oumar El-Shangiti, Hiba Zayed, Mohamedou Cheikh Tourad, Rahaf Alhamouri, Rwaa Assi, Aisha Alraeesi, Hour Mohamed, Fakhraddin Alwajih, Abdelrahman Mohamed, Abdellah {El Mekki}, El Moatez Billah Nagoudi, Benelhadj Djelloul Mama Saadia, Hamzah A. Alsayadi, Walid Al-Dhabyani, Sara Shatnawi, Yasir Ech-chammakhy, Amal Makouar, Yousra Berrachedi, Mustafa Jarrar, Shady Shehata, Ismail Berrada and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1211},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1211}
}

References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Amr Keleg, Abdelrahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. Nadi 2024: The fifth nuanced arabic dialect identification shared task. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 709– 728.
  2. Muhammad Abdul Abdul-Mageed, Chiyu Zhang, Abdelrahim Elmadany, and Lyle Ungar. 2020. Toward micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876. find
  3. Abdelrahman Ahmed, Yasser Hifny, Khaled Shaalan, and Sergio Toral. 2019. End-to-end lexicon free arabic speech recognition using recurrent neural networks. In Computational Linguistics, Speech And Image Processing For Arabic Language, pages 231– 248. World Scientific. ↗
  4. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279– 284. IEEE.
  5. Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, James Glass, Steve Renals, and Khalid Choukri. 2019. The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1026–1033. IEEE. find
  6. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic mgb-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316– 322. IEEE.
  7. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670. ↗
  8. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296. ↗
  9. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  10. Mohammed Bakheet. 2021. Improving speech recognition for arabic language using low amounts of labeled data. find
  11. Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596. ↗
  12. Martijn Bartelds, Nay San, Bradley McDonnell, Dan Jurafsky, and Martijn Wieling. 2023. Making more of little data: Improving low-resource automatic speech recognition using data augmentation. arXiv preprint arXiv:2305.10951. ↗
  13. Hervé Bredin and Antoine Laurent. 2021. End-to-end speaker segmentation for overlap-aware resegmentation. In Proc. Interspeech 2021, Brno, Czech Republic. find
  14. Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2020. pyannote.audio: neural building blocks for speaker diarization. In ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain. find
  15. Keith Brown. 2005. Encyclopedia of language and linguistics, volume 1. Elsevier. find
  16. David Cohen, Mikael Lindvall, and Patricia Costa. 2004. An introduction to agile methods. Adv. Comput., 62(03):1–66. ↗
  17. George E Dahl, Dong Yu, Li Deng, and Alex Acero. 2011. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on audio, speech, and language processing, 20(1):30–42. ↗
  18. Yousif A El-Imam. 2004. Phonetization of arabic: rules and algorithms. Computer Speech & Language, 18(4):339–373. ↗
  19. Nizar Habash. 2022. Arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 9–10. ↗
  20. Amir Hussein, Shinji Watanabe, and Ahmed Ali. 2022. Arabic speech recognition by end-to-end, modular systems and human. Computer Speech & Language, 71:101272. ↗
  21. Sameer Khurana and Ahmed Ali. 2016. Qcri advanced transcription system (qats) for the arabic multi-dialect broadcast media recognition: Mgb-2 challenge. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 292–298. IEEE. find
  22. Marieke Meelen, Alexander O’neill, and Rolando Coto-Solano. 2024. End-to-end speech recognition for endangered languages of nepal. In Proceedings of the Seventh Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 83–93. find
  23. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. Qasr: Qcri aljazeera speech resource–a large scale annotated arabic speech corpus. arXiv preprint arXiv:2106.13000. ↗
  24. Tobi Olatunji, Tejumade Afonja, Aditya Yadavalli, Chris Chinenye Emezue, Sahib Singh, Bonaventure FP Dossou, Joanne Osuchukwu, Salomey Osei, Atnafu Lambebo Tonja, Naome Etori, et al. 2023. Afrispeech-200: Pan-african accented speech dataset for clinical and general domain asr. Transactions of the Association for Computational Linguistics, 11:1669–1685. ↗
  25. Joseph Olive, Caitlin Christianson, and John McCary. 2011. Handbook of natural language processing and machine translation: DARPA global autonomous language exploitation. Springer Science & Business Media. ↗
  26. David S Pallett. 2003. A look at nist’s benchmark asr tests: past, present, and future. In 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No. 03EX721), pages 483–488. IEEE. find
  27. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  28. Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, et al. 2023. Stt4sg-350: A speech corpus for all swiss german dialect regions. arXiv preprint arXiv:2305.18855. ↗
  29. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516. ↗
  30. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. ↗
  31. Hagen Soltau, George Saon, Brian Kingsbury, Hong-Kwang Jeff Kuo, Lidia Mangu, Daniel Povey, and Ahmad Emami. 2009. Advances in arabic speech transcription at ibm under the darpa gale program. IEEE Transactions on Audio, Speech, and Language Processing, 17(5):884–894. ↗
  32. Peter Sullivan, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2023. On the robustness of arabic speech dialect identification. arXiv preprint arXiv:2306.03789. ↗
  33. Bashar Talafha, Abdul Waheed, and Muhammad Abdul-Mageed. 2023. N-shot benchmarking of whisper on diverse arabic speech recognition. arXiv preprint arXiv:2306.02902. ↗
  34. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020. Label studio: Data labeling software. Open source software available from https://github.com/heartexlabs/labelstudio, 2022. ↗
  35. Atnafu Lambebo Tonja, Fazlourrahman Balouchzahi, Sabur Butt, Olga Kolesnikova, Hector Ceballos, Alexander Gelbukh, and Thamar Solorio. 2024. Nlp progress in indigenous latin american languages. arXiv preprint arXiv:2404.05365. ↗
  36. Dave Uthus, Garrett Tanzer, and Manfred Georg. 2024. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems, 36. find
  37. Janet CE Watson. 2007. The phonology and morphology of Arabic. OUP Oxford. find
  38. Brian A Weiss, Craig Schlenoff, Gregory A Sanders, Michelle Potts Steves, Sherri L Condon, Jon Phillips, and Dan Parvaz. 2008. Performance evaluation of speech translation systems. In LREC. find
  39. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. 2023. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037. ↗
  40. Liang Zhao and Eleanor Chodroff. 2022. The mandi corpus: A spoken corpus of mandarin regional dialects. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1985– 1990. find
  41. Taha Zouhair. 2021. Automatic speech recognition for low-resource languages using wav2vec2: Modern standard arabic (msa) as an example of a low-resource language. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talafha-2024-casablanca-data-models-multidialectal-arabic/