Elamur Research⌕
☰
home / papers / 2026 / EDRAC: Benchmarking Arabic Dialect Reading Comprehension

EDRAC: Benchmarking Arabic Dialect Reading Comprehension

Summary

Dialectal Arabic has far fewer resources than Modern Standard Arabic. Existing dialect reading-comprehension tests are almost all multiple-choice and built from translated or social-media text. EDRAC fills this gap with an open-ended, generative QA benchmark for five dialects: Egyptian, Moroccan, Emirati, Syrian and Saudi.

The passages come from YouTube subtitles of spoken content. They were processed with speaker diarization, alignment, text restoration and extraction, then corrected by native speakers. Gemini 2.5 pro drafted the question–answer pairs in repeated rounds, an LLM judge scored each round, and a professional language service provider checked every pair. The release has 499 passages and 4,977 human-validated pairs.

The authors tested Arabic-centric and multilingual LLMs with lexical and semantic metrics plus a human study. Models often give answers that are correct in meaning but not in the dialect, and standard metrics miss this. The benchmark therefore shows that metric choice can distort model rankings, which matters for anyone evaluating generation in non-standard language varieties.

Key points

  • Size and license: 499 passages and 4,977 QA pairs, about 1,000 per dialect, released on Hugging Face (MBZUAI/EDRAC) under CC BY-NC-SA 4.0 for evaluation only.
  • Source data:
    • Native speakers proposed YouTube channels, and videos from June 2024 to January 2026 were used.
    • About 150 videos per dialect were gathered against a target of 100, to allow filtering.
    • Each passage is the first six minutes of a video, roughly 473 tokens.
    • Speaker diarization used NVIDIA NeMo.
  • Ranking depends on the metric: GPT-5.4 ranked best overall but fell to joint 5th, tied with Qwen2.5-7B, under CAMeLBERTScore-F1. Hala-9B was the weakest model.
  • Human evaluation: 40 outputs per dialect (GPT-5.4 and Hala-9B) were rated for accuracy and naturalness.
    • Automatic scores: ROUGE 8–13%, BERTScore 79–85%.
    • Human accuracy: 82–100%.
    • Human naturalness (dialectness): only 5–50%, with Emirati lowest at 5%.

Interesting ideas

  • I1. Judge an answer on two separate axes, whether its content is right and whether its form matches the target variety, because one similarity score hides fluent but off-register output. — For Mova (back): Generated text could get two separate checks, one for meaning against the graph and one for register or style, so a correct but unnatural sentence is still flagged.
  • I2. Human raters saw only the reference answer, not the passage, so wording in the source could not prime them. — For Mova (learn): When people or an LLM teacher grade Mova's answers, hiding the source text forces them to judge meaning equivalence rather than surface overlap.
  • I3. Draft QA pairs with an LLM, have another LLM pass critique and refine them over several rounds, and send only the survivors to human checkers, which cuts the cost of human annotation. — For Mova (answer): New question sets for fables and tales could be built this way, with a cheap judge loop before any manual review.
  • I4. Rank the same systems under several metrics and treat big rank jumps as a warning that a metric is measuring the wrong thing. — For Mova (learn): Mova's evaluations could report rankings across multiple metrics and flag cases where they disagree, much like a negative control on the metric itself.
  • I5. Collect about 50% more raw material than the target, then filter for quality, toxicity and missing transcripts, so the final size survives strict cleaning. — For Mova (learn): Silver-data harvesting could over-sample in the same way so that witness-parser disagreement filtering still leaves enough training sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.01113 · arXiv 2609.01113

@misc{mokh2026edrac,
  title = {{EDRAC: Benchmarking Arabic Dialect Reading Comprehension}},
  author = {Mokh, Noor Abo and Chirkunov, Kirill and Lynn, Teresa and Habash, Nizar and Marzouk, Reham and Altakrori, Malik H. and Samih, Younes and Odeh, Muhammed Abu and Rabih, Nour and Alshahrani, Rahaf and Alshehhi, Hamad and Al-Ali, Hamdan and Almahri, Muhra and Hassan, Besher and Anwar, Mohamed and Freihat, Abed Alhakim and Nakov, Preslav and Aji, Alham Fikri},
  year = {2026},
  doi = {10.48550/arxiv.2609.01113},
  eprint = {2609.01113},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.01113}
}

References

48 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. ArabicaQA: A comprehensive dataset for Arabic question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, pages 2049–2059, New York, NY, USA. Association for Computing Machinery. find our card
  2. Mouath Abu-Daoud, Leen Kharouf, Omar El Hajj, Dana Samad, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash, and Farah Shamout. 2026. MedAraBench: Large-scale Arabic medical question answering dataset and benchmark. In International Conference on Learning Representations, volume 2026 of ICLR ’26, pages 13469–13496, Rio de Janeiro, Brazil. find our card
  3. Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, and Mirella Lapata. 2023. QAmeleon: Multilingual QA with only 5 examples. Transactions of the Association for Computational Linguistics, 11:1754–1771. find
  4. Maryam Khalifa Al Ali and Hanan Aldarmaki. 2024. Mixat: A data set of bilingual Emirati-English speech. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 222–226, Torino, Italia. ELRA and ICCL. find
  5. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2023. MASC: Massive arabic speech corpus. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1006–1013. find
  6. Rawan Al-Matham, Kareem Darwish, Raghad Al-Rasheed, Waad Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah Alzahrani, Eman AlBilali, Nizar Habash, Abdelrahman El-Sheikh, Muhammad Elmallah, Haonan Li, Hamdy Mubarak, Mohamed Anwar, Zaid Alyafeai, and 24 others. 2025. BALSAM: A platform for benchmarking Arabic large language models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, Suzhou, China. Association for Computational Linguistics. find
  7. Aamer Aljagthami, Mohammed Banabila, Musab Alshehri, Mohammed Kabini, and Mohammad D. Alahmadi. 2025. Evaluating large language models for code translation: Effects of prompt language and prompt design. arXiv preprint arXiv:2509.12973. ↗
  8. Omar Alkaabi, Ahmed Alzubaidi, Hamza Alobeidli, Shaikha Alsuwaidi, Mohammed Alyafeai, Leen AlQadi, Basma El Amel Boussaha, and Hakim Hacid. 2026. Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models. find
  9. Sultan Alrowili and K Vijay-Shanker. 2023. ArTrivia: Harvesting Arabic Wikipedia to build a new Arabic question answering dataset. In Proceedings of ArabicNLP 2023, pages 191–207, Singapore (Hybrid). Association for Computational Linguistics. find
  10. Malik H. Altakrori, Nizar Habash, Teresa Lynn, Younes Samih, Abed Alhakim Freihat, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Preslav Nakov, and Alham Fikri Aji. 2026. DialectalArabicMMLU: Benchmarking dialectal capabilities in Arabic and multilingual language models. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 3199–3219, Palma, Mallorca, Spain. European Language Resources Association (ELRA). find our card
  11. Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-Domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520–534, Online. Association for Computational Linguistics. ↗ our card
  12. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on Twitter. In International Conference on Applications of Natural Language to Information Systems, pages 57–64. Springer. find
  13. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: A parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics. find our card
  14. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  15. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3). find our card
  16. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  17. Abdelghani Dahou, Abdelhalim Hafedh Dahou, Mohamed Amine Cheragui, Amin Abdedaiem, Mohammed A. A. Al-Qaness, Mohamed Abd Elaziz, Ahmed A. Ewees, and Zhonglong Zheng. 2025. A survey on dialect Arabic processing and analysis: Recent advances and future trends. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 24(8). find
  18. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the Arab world. Commun. ACM, 64(4):72–81. Krishno Dey, Prerona Tarannum, Md. find
  19. Arid Hasan, Imran Razzak, and Usman Naseem. 2024. Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings. Preprint, arXiv:2410.13153. ↗
  20. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359–369, Atlanta, Georgia. Association for Computational Linguistics. find our card
  21. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Octopus: A multitask model and toolkit for Arabic natural language generation. In Proceedings of ArabicNLP 2023, pages 232–243, Singapore (Hybrid). Association for Computational Linguistics. find
  22. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing, 8(4). find
  23. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find
  24. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
  25. Injy Hamed, Ngoc Thang Vu, and Slim Abdennadher. 2020. ArzEn: A speech corpus for code-switched Egyptian Arabic-English. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4237–4246, Marseille, France. European Language Resources Association. find
  26. Hunzalah Hassan Bhatti and Firoj Alam. 2026. Beyond MCQ: An open-ended Arabic cultural QA benchmark with dialect variants. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 5215–5231, Palma, Mallorca, Spain. ELRA Language Resource Association. find our card
  27. Mariam E. Hassib, Nagwa El-Makky, and Marwan Torki. 2025. Open-domain Arabic conversational question answering with question rewriting. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 84–96, Suzhou, China. Association for Computational Linguistics. find
  28. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543. ↗ our card
  29. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2). find our card
  30. Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2024. Explainable multi-hop question generation: An end-to-end approach without intermediate question labeling. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6855–6866, Torino, Italia. ELRA and ICCL. find
  31. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  32. Abdullah Khered, Youcef Benkhedda, and Riza Batista-Navarro. 2025. Dial2MSA-Verified: A multi-dialect Arabic social media dataset for neural machine translation to Modern Standard Arabic. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 50–62, Abu Dhabi, UAE. Association for Computational Linguistics. find
  33. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine-grained evaluation capability in language models. Preprint, arXiv:2310.08491. ↗ our card
  34. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  35. Zefeng Lin, Weidong Chen, Yan Song, and Yongdong Zhang. 2024. Prompting few-shot multi-hop question generation via comprehending type-aware semantics. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3730–3740, Mexico City, Mexico. Association for Computational Linguistics. find
  36. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics. find
  37. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. find our card
  38. OpenRouter. 2026. OpenRouter: Unified API for Large Language Models. https://openrouter.ai. Accessed: 2026-05-22. ↗
  39. Soufiyan Ouali and Said El Garouani. 2026. MedQA-MA: A Moroccan Arabic medical question-answering dataset for virtual healthcare assistants and large language models. Data in Brief, 65:112537. find
  40. Aurelien Pellet, Marie Anna Puren, and Julien Perez. 2026. HistoriQA-ThirdRepublic: Multi-hop question answering corpus for historical research, parliamentary debates from the French third republic (1870-1940). In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 207–223, Palma, Mallorca, Spain. European Language Resources Association (ELRA). find our card
  41. José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and Andr’e F. T. Martins. 2025. M-prometheus: A suite of open multilingual LLM judges. In Proceedings of the Second Conference on Language Modeling. find
  42. John W. Ratcliff and David E. Metzener. 1988. Pattern matching: The gestalt approach. Dr. Dobb’s Journal. find
  43. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. ↗ our card
  44. Manuela Sanguinetti, Cristina Bosco, Lauren Cassidy, Özlem Çetinoğlu, Alessandra Teresa Cignarella, Teresa Lynn, Ines Rehbein, Josef Ruppenhofer, Djamé Seddah, and Amir Zeldes. 2020. Treebanking user-generated content: A proposal for a unified representation in Universal Dependencies. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5240–5250, Marseille, France. European Language Resources Association. find
  45. Abdellah Hamouda Sidhoum, M’hamed Mataoui, Faouzi Sebbak, and Kamel Smaïli. 2022. ACQAD: A Dataset for Arabic Complex Question Answering. In Atlantis Press Proceedings., Boumerdès, Algeria. find
  46. Romy A.N. van Drie, Roos M. Bakker, Daan L. Di Scala, and Maaike de Boer. 2026. QuALA-NL: Question & answer with legal attribution in Dutch. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 674– 684, Palma, Mallorca, Spain. European Language Resources Association (ELRA). find
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations, ICLR ’20. find
  48. Inès Zribi, Mariem Ellouze, Bilel Hamrouni, and Lamia H. Belguith. 2015. Spoken Tunisian Arabic corpus "STAC": Transcription and annotation. Research in Computing Science, 90:123–135. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mokh-2026-edrac-benchmarking-arabic-dialect-reading/