Elamur Research⌕
☰
home / papers / 2024 / Open Universal Arabic ASR Leaderboard

Open Universal Arabic ASR Leaderboard

Summary

Arabic speech recognition is moving toward single models that are expected to handle every dialect, but earlier benchmarks tested only a few models, often on one dataset or on corpora covering just one dialect. The authors build a continuously updated, open leaderboard, modelled on the English Open ASR Leaderboard, for general-purpose open-source Arabic ASR systems. They run zero-shot inference over four model families: Whisper variants, NVIDIA's Conformer-CTC, community fine-tuned self-supervised models (wav2vec 2.0, XLS-R, MMS, w2v-BERT, HuBERT) and Seamless M4T. Evaluation uses only multi-dialectal test sets, and models are ranked by average WER, with CER also reported. Beyond accuracy, the paper examines robustness, speaker adaptation, inference speed and memory use. The Conformer-CTC model paired with a 4-gram language model comes out on top, ahead of whisper-large-v3. The benchmark gives the community a shared reference point and a common evaluation protocol for dialect-general Arabic ASR.

Key points

  • The benchmark has five test sets: SADA (Test), Common Voice 18.0 (Test), MASC Test-Clean, MASC Test-Noisy and MGB-2 (Test). All of them are multi-dialectal.
  • The leaderboard has 14 entries. nvidia/conformer-ctc-large-ar ranks first at 25.71% average WER with the LM and second at 27.46% with greedy decoding; whisper-large-v3 is third at 29.87%.
  • The last-placed entry is speechbrain/wav2vec2-large-ft at 60.15% average WER. whisper-large-v3-turbo (33.30%) scores slightly below v3, a modest cost for its 4 decoder layers instead of 32.
  • Seamless M4T supports three Arabic varieties (MSA, Moroccan, Egyptian). It was run once per setting, and only its best score was reported.
  • All results are zero-shot: no model was adapted to the benchmark data.

Interesting ideas

  • I1. The authors refuse to judge dialect ability by comparing scores across separate single-dialect corpora, because recording conditions and other corpus traits confound the comparison; they test only on mixed-dialect sets instead. — For Mova (tree): When measuring how well a parser handles a UD dialect or genre, compare within one treebank that mixes the varieties, rather than across treebanks whose annotation habits differ.
  • I2. The same model appears twice on the leaderboard, once with greedy decoding and once with a language-model beam search, so the gain from the decoding add-on is directly visible. — For Mova (tree): Report perceptron and gparse both bare and with extra constraints or rescoring as separate rows, so each component's contribution can be explained.
  • I3. MASC is scored as separate clean and noisy splits, a cheap robustness probe that keeps the content distribution constant while degrading the input. — For Mova (tokens): Build a noisy copy of a tagging test set (typos, dropped punctuation, casing changes) to measure how much degradation alone costs the tokenizer and tagger.
  • I4. Models are ranked by an average over several heterogeneous test sets rather than a single headline corpus, which discourages overfitting to one benchmark. — For Mova (learn): Score each training round on an average across several held-out sources, such as school tests, fables and word problems, so a gain on one source cannot hide regressions elsewhere.
  • I5. When a system needs a configuration choice (here, the language setting), the paper reports its best configuration; this is convenient but slightly favours that system. — For Mova (answer): If readers are run with several settings, record which setting was picked and how, ideally on a dev split, so oracle selection does not inflate results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech & spoken language · Evaluation & benchmarks · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.13788 · arXiv 2412.13788

@misc{wang2024open,
  title = {{Open Universal Arabic ASR Leaderboard}},
  author = {Wang, Yingzhi and Alhmoud, Anas and Alqurishi, Muhammad},
  year = {2024},
  doi = {10.48550/arxiv.2412.13788},
  eprint = {2412.13788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.13788}
}

References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Dhouib, A. Othman, O. El Ghoul, M. K. Khribi, and A. Al Sinani, “Arabic automatic speech recognition: a systematic literature review,” Applied Sciences, vol. 12, no. 17, p. 8898, 2022. find
  2. A. A. Abdelhamid, H. A. Alsayadi, I. Hegazy, and Z. T. Fayed, “End-to-end arabic speech recognition: A review,” in Proceedings of the 19th Conference of Language Engineering (ESOLEC’19), Alexandria, Egypt, 2020, pp. 26–30. find
  3. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 279–284. find our card
  4. S. Alharbi, A. Alowisheq, Z. Tüske, K. Darwish, A. Alrajeh, A. Alrowithi, A. B. Tamran, A. Ibrahim, R. Aloraini, R. Alnajim et al., “Sada: Saudi audio dataset for arabic,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 286–10 290. find
  5. M. Al-Fetyani, M. Al-Barham, G. Abandah, A. Alsharkawi, and M. Dawas, “Masc: Massive arabic speech corpus,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1006–1013. find
  6. H. Mubarak, A. Hussein, S. A. Chowdhury, and A. Ali, “Qasr: Qcri aljazeera speech resource–a large scale annotated arabic speech corpus,” arXiv preprint arXiv:2106.13000, 2021. ↗
  7. B. Talafha, K. Kadaoui, S. M. Magdy, M. Habiboullah, C. M. Chafei, A. O. El-Shangiti, H. Zayed, R. Alhamouri, R. Assi, A. Alraeesi et al., “Casablanca: Data and models for multidialectal arabic speech recognition,” arXiv preprint arXiv:2410.04527, 2024. ↗
  8. A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Machine Learning. PMLR, 2022, pp. 1298–1312. find
  9. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020. find our card
  10. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021. find our card
  11. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. ↗ our card
  12. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518. find our card
  13. T. Wolf, “Huggingface’s transformers: State-of-the-art natural language processing,” arXiv preprint arXiv:1910.03771, 2019. ↗ our card
  14. O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al., “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577, 2019. ↗
  15. M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y. Wang, P. Mousavi, L. Della Libera, A. Ploujnikov et al., “Open-source conversational ai with speechbrain 1.0,” Journal of Machine Learning Research, vol. 25, no. 333, pp. 1–11, 2024. find
  16. B. Talafha, A. Waheed, and M. Abdul-Mageed, “N-shot benchmarking of whisper on diverse arabic speech recognition,” arXiv preprint arXiv:2306.02902, 2023. ↗ our card
  17. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” arXiv preprint arXiv:2006.13979, 2020. ↗
  18. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019. ↗ our card
  19. A. Ali, S. Vogel, and S. Renals, “Speech recognition challenge in the wild: Arabic mgb-3,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 316–322. find our card
  20. A. Ali, S. Shon, Y. Samih, H. Mubarak, A. Abdelali, J. Glass, S. Renals, and K. Choukri, “The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 1026–1033. find
  21. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 798–805. find our card
  22. A. Abdelali, H. Mubarak, S. Chowdhury, M. Hasanain, B. Mousi, S. Boughorbel, S. Abdaljalil, Y. El Kheir, D. Izham, F. Dalvi et al., “Larabench: Benchmarking arabic ai with large language models,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 487–520. find our card
  23. V. Srivastav, S. Majumdar, N. Koluguri, A. Moumen, S. Gandhi et al., “Open automatic speech recognition leaderboard,” https://huggingface.co/spaces/hf-audio/open asr leaderboard, 2023. ↗
  24. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376. find
  25. A. Graves, “Sequence transduction with recurrent neural networks,” arXiv preprint arXiv:1211.3711, 2012. ↗ our card
  26. V. Noroozi, S. Majumdar, A. Kumar, J. Balam, and B. Ginsburg, “Stateful conformer with cache-based inference for streaming automatic speech recognition,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 041–12 045. find
  27. N. Halabi et al., “Arabic speech corpus,” Oxford Text Archive Core Collection, 2016. find
  28. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al., “Scaling speech technology to 1,000+ languages,” Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024. find our card
  29. Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 244–250. find
  30. L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffman et al., “Seamlessm4t-massively multilingual & multimodal machine translation,” arXiv preprint arXiv:2308.11596, 2023. ↗ our card
  31. S. A. Chowdhury, A. Hussein, A. Abdelali, and A. Ali, “Towards one model to rule all: Multilingual strategy for dialectal code-switching arabic asr,” arXiv preprint arXiv:2105.14779, 2021. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/