Elamur Research⌕
☰
home / papers / 2025 / Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps

Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps

Paper · 2025 · Ahmed Alzubaidi, Shaikha Alsuwaidi, Basma El Amel Boussaha et al.

Summary

Arabic LLM evaluation is spread across dozens of benchmarks that differ in how they were built, how large they are, and whether others can reproduce them. Until this survey, no review had mapped them systematically. The authors examine more than 40 Arabic benchmarks and sort them into four groups: Knowledge (general, STEM, law, medicine, religion, poetry), NLP Tasks (from single-task sets up to multi-task suites), Culture and Dialects, and Target-Specific (safety, hallucination, instruction-following, vision). They compare three ways of building a benchmark: collecting native Arabic material, translating English benchmarks, and generating items synthetically with LLMs. Each is weighed for authenticity, scale and cost. They also cover metrics, LLM-as-judge practice and public leaderboards. The main finding is that the field now has many kinds of benchmarks but leaves several areas thin: change over time, multi-turn dialogue, code-switching, rare dialects and pragmatics. Translated sets often carry cultural mismatch. The paper ends with eight recommendations and a public repository. It is useful to anyone designing honest evaluations for a language other than English.

Key points

  • The taxonomy has four top-level categories. The benchmarks, links and code are collected in the GitHub repository tiiuae/Arabic-LLM-Benchmarks.
  • Translated and native knowledge sets are of similar size:
    • MMLU_ar has 14,079 human-translated MCQs.
    • ArabicMMLU has 14,575 native MCQs drawn from school and university exams across Arab countries.
    • 3LM has 3,151 items in native, synthetic (Qwen3-235B-A22B via Yourbench) and translated-code parts.
    • GATmath and GATLc together hold about 16k items.
  • Large suites:
    • LAraBench covers 33 tasks over 61 datasets.
    • BALSAM covers 78 tasks with 52K samples and keeps its test sets private.
    • OALL v2 dropped translated sets in favour of native ones such as ArabicMMLU, AraTrust and MadinahQA.
    • The AraGen leaderboard uses the 3C3H judge rubric (Correctness, Completeness, Conciseness, Helpfulness, Honesty, Harmlessness).
  • Benchmark size ranges from under 500 items (Qiyas, EXAMS_ar) to over 2 million (Dolphin). The authors warn that small sets give weak signals and that unweighted averages across sets can mislead.
  • The eight recommendations include:
    • native content with cultural review;
    • full code release;
    • blind test sets against contamination;
    • standard prompting and MCQ formatting;
    • validating judges against human ratings;
    • wider dialect coverage;
    • stronger quality control;
    • dynamic, time-aware evaluation.

Interesting ideas

  • I1. Keep a sealed test split that no training or generation pipeline ever touches, because contamination quietly inflates scores as corpora grow.
  • I2. When an LLM both writes test items and resembles the model being graded, the evaluation becomes circular; generator, judge and examinee should be kept apart.
  • I3. Do not average scores across benchmarks of very different sizes without weighting, since tiny sets are noisy and easy to overfit.
  • I4. Material translated from another language can carry the source culture's assumptions; native-sourced items with a cultural review test real competence more faithfully.
  • I5. The survey names sarcasm and indirect speech as almost unbenchmarked, which shows that pragmatic skill needs its own targeted diagnostic sets.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Hold back a locked evaluation slice that the silver-data witnesses and LLM teacher tiers never see, so self-repair rules and training gains are measured on genuinely unseen text.
  • I2 (learn): Questions or gold labels produced by an LLM teacher should not come from the same model family that produced the silver data being scored, which acts as a built-in negative control.
  • I3 (answer): Report school-test, word-problem and fable-QA results separately, with item counts and weighted totals, so a strong score on a small fable set cannot hide a weak maths result.
  • I4 (answer): Build reading-comprehension tests from texts originally written in English (native fables, real school exams) rather than adapted ones, so world-of-the-text reasoning is not disturbed by translation artefacts.
  • I5 (frames): Write a small dedicated test of indirect speech acts, implicature and irony for crate prag, scoring the explained frame rather than only the final label.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Ahmed Alzubaidi, Shaikha Alsuwaidi, Basma El Amel Boussaha, Leen AlQadi, Omar Alkaabi, Mohammed Alyafeai, Hamza Alobeidli and Hakim Hacid. 2025. Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps. arXiv:2510.13430.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.13430 · arXiv 2510.13430

@misc{alzubaidi2025evaluating,
  title = {{Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps}},
  author = {Ahmed Alzubaidi, Shaikha Alsuwaidi, Basma {El Amel Boussaha}, Leen AlQadi, Omar Alkaabi, Mohammed Alyafeai, Hamza Alobeidli and Hakim Hacid},
  year = {2025},
  doi = {10.48550/arxiv.2510.13430},
  eprint = {2510.13430},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.13430}
}

References

77 references, 55 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin. 2025. Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations. Preprint, arXiv:2503.07833. ↗
  2. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  3. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. Nadi 2024: The fifth nuanced arabic dialect identification shared task. arXiv preprint arXiv:2407.04910. aiastrolabe. 2025. Redteaming frontier llms with ai astrolabe arabic safety index (asas). Accessed: Oct. 3, 2025. ↗
  4. Shahad Al-Khalifa and Hend Al-Khalifa. 2024. The qiyas benchmark: Measuring chatgpt mathematical and language understanding in arabic. arXiv preprint arXiv:2407.00146. ↗
  5. Shahad Al-Khalifa, Nadir Durrani, Hend Al-Khalifa, and Firoj Alam. 2025. The landscape of arabic large language models. Communications of the ACM. ↗
  6. Nashwan Ahmed Al-Majmar, Hezam Gawbah, and Akram Alsubari. 2024. Ahd: Arabic healthcare dataset. Data in Brief, 56:110855. ↗
  7. Rawan Al-Matham, Kareem Darwish, Raghad Al-Rasheed, Waad Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah Alzahrani, Eman alBilali, Nizar Habash, Abdelrahman El-Sheikh, Muhammad Elmallah, Haonan Li, Hamdy Mubarak, Mohamed Anwar, Zaid Alyafeai, and 24 others. 2025. Balsam: A platform for benchmarking arabic large language models. Preprint, arXiv:2507.22603. ↗
  8. Renad Al-Monef, Hassan Alhuzali, Nora Alturayeif, and Ashwag Alasmari. 2025. Absher: A benchmark for evaluating large language models’ understanding of saudi dialects. Preprint, arXiv:2507.10216. ↗
  9. Aisha Alansari and Hamzah Luqman. 2025. Arahallueval: A fine-grained hallucination evaluation framework for arabic llms. Preprint, arXiv:2509.04656. ↗
  10. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OS-ACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation@ LREC-COLING 2024, pages 50–56.
  11. Safa AlBallaa, Nora AlTwairesh, Abdulmalik AlSalman, and Sultan Alfarhood. 2025. Gatmath and gatlc: Comprehensive benchmarks for evaluating arabic large language models. PLoS One, 20(9):e0329129. ↗
  12. Hayfa A Aleid and Aqil M Azmi. 2025. Hajj-fqa: A benchmark arabic dataset for developing question-answering systems on hajj fatwas: H. aleid and a. azmi. Journal of King Saud University Computer and Information Sciences, 37(6):135. ↗
  13. Wafa Alghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025. Fann or flop: A multigenre, multiera benchmark for arabic poetry understanding in llms. arXiv preprint arXiv:2505.18152. ↗
  14. Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2024. Aratrust: An evaluation of trustworthiness for llms in arabic. Preprint, arXiv:2403.09017. ↗
  15. Mohamed Alkaoud. 2024. A bilingual benchmark for evaluating large language models. PeerJ Computer Science, 10:e1893. ↗
  16. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics.
  17. Rana Alshaikh, Israa Alghanmi, and Shelan Jeawak. 2025. Aratable: Benchmarking llms’ reasoning and understanding of arabic tabular data. Preprint, arXiv:2507.18442. ↗
  18. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025a. Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics.
  19. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025b. Palmx 2025: The first shared task on benchmarking llms on arabic and islamic culture. Preprint, arXiv:2509.02550. ↗
  20. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of arabic multimodal large language models and benchmarks. Preprint, arXiv:2403.01031. ↗
  21. Anthropic. 2024. Introducing the next generation of claude. find
  22. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  23. Yasser Ashraf, Yuxia Wang, Bin Gu, Preslav Nakov, and Timothy Baldwin. 2025. Arabic dataset for llm safeguard evaluation. Preprint, arXiv:2410.17040. ↗
  24. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732. ↗
  25. Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, and 8 others. 2022. Promptsource: An integrated development environment and repository for natural language prompts. Preprint, arXiv:2202.01279. ↗
  26. Adil Bahaj and Mounir Ghogho. 2025. Mizanqa: Benchmarking large language models on moroccan legal question answering. arXiv preprint arXiv:2508.16357. ↗
  27. Basma El Amel Boussaha, Leen AlQadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, and Hakim Hacid. 2025. 3lm: Bridging arabic, stem, and code through benchmarking. arXiv preprint arXiv:2507.15850. ↗
  28. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374. ↗
  29. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Preprint, arXiv:2003.05002. ↗
  30. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. ↗
  31. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv preprint arXiv:2505.03427. ↗
  32. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. Preprint, arXiv:2305.14387. ↗
  33. Ali El Filali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024a. Open arabic llm leaderboard. https://huggingface.co/ spaces/OALL/Open-Arabic-LLM-Leaderboard. ↗
  34. Ali El Filali, Manel ALOUI, Tarique Husaain, Ahmed Alzubaidi, Basma El Amel Boussaha, Ruxandra Cojocaru, Clémentine Fourrier, Nathan Habib, and Hakim Hacid. 2025. The open arabic llm leaderboard 2. https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard. ↗
  35. Ali El Filali, Neha Sengupta, Abouelseoud, Preslav Nakov, and Clémentine Fourrier. 2024b. Rethinking llm evaluation with 3c3h: Aragen benchmark and leaderboard. https://https://huggingface.co/ blog/leaderboard-3c3h-aragen. ↗
  36. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Orca: A challenging benchmark for arabic language understanding. Preprint, arXiv:2212.10758. ↗
  37. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. The language model evaluation harness. find
  38. Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad S. Khan, Salman Khan, and Rao M. Anwer. 2024. Camel-bench: A comprehensive arabic lmm benchmark. Preprint, arXiv:2410.18976. ↗
  39. Sara Ghaboura, Ketan More, Wafa Alghallabi, Omkar Thawakar, Jorma Laaksonen, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025. Arb: A comprehensive arabic multimodal reasoning benchmark. Preprint, arXiv:2505.17021. ↗
  40. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  41. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Preprint, arXiv:2308.11462. ↗
  42. Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation. find
  43. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. Exams: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. arXiv preprint arXiv:2011.03080. ↗
  44. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  45. Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah Al-Shamlan, Omar Knio, and George Turkiyyah. 2024. Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models. arXiv preprint arXiv:2408.07983. ↗
  46. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. Acegpt, localizing large language models in arabic. Preprint, arXiv:2309.12053. ↗
  47. Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu. 2025. A survey on large language models with multilingualism: Recent advances and new frontiers. Preprint, arXiv:2405.10936. ↗
  48. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2025. Revisiting common assumptions about arabic dialects in nlp. arXiv preprint arXiv:2505.21816. ↗
  49. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. ↗
  50. Anis Koubaa, Adel Ammar, Lahouari Ghouti, Omar Najar, and Serry Sibaee. 2024. Arabiangpt: Native arabic gpt-based large language model. Preprint, arXiv:2402.15313. ↗
  51. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. Preprint, arXiv:2109.07958. ↗
  52. Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Taher Abdelfadil, Shady Shehata, and Muhammad Abdul-Mageed. 2025. JAWAHER: A multidialectal dataset of Arabic proverbs for LLM benchmarking. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12320–12341, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  53. Malak Mashaabi, Shahad Al-Khalifa, and Hend Al-Khalifa. 2024. A survey of large language models for arabic language and its dialects. arXiv preprint arXiv:2410.20238. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. ↗
  54. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.
  55. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem M. Hajj. 2019. Neural arabic question answering. CoRR, abs/1906.05394. ↗
  56. Hamdy Mubarak, Hend Al-Khalifa, and Khaloud Suliman Alkhalefah. 2024. Halwasa: Quantify and analyze hallucinations in large language models: Arabic as a case study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8008–8015, Torino, Italia. ELRA and ICCL.
  57. Omartificial-Intelligence-Space. 2025. Arabic gsm8k: Arabic grade school math dataset. https://huggingface.co/datasets/ Omartificial-Intelligence-Space/ Arabic-gsm8k-v2. ↗
  58. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023a. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. ↗
  59. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  60. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023b. Crosslingual generalization through multitask finetuning. Preprint, arXiv:2211.01786. ↗
  61. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. arXiv preprint arXiv:2501.00559. ↗
  62. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S. Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader, and Anis Koubaa. 2025. Towards inclusive Arabic LLMs: A culturally aligned benchmark in Arabic large language model evaluation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 387–401, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  63. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023. Dolphin: A challenging and diverse benchmark for Arabic NLG. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1404–1422, Singapore. Association for Computational Linguistics.
  64. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2025. A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5):1–72. ↗
  65. OpenAI. 2023. Chatgpt. https://chat.openai.com/ chat. Accessed: 2 October 2025. ↗
  66. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. Preprint, arXiv:2409.12623. ↗
  67. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. ↗
  68. Haneh Rhel and Dmitri Roussinov. 2025. Large language models and arabic content: a review. arXiv preprint arXiv:2505.08004. ↗
  69. Nathaniel R. Robinson, Shahd Abdelmoneim, Kelly Marchisio, and Sebastian Ruder. 2025. Al-qasida: Analyzing llm quality and accuracy systematically in dialectal arabic. In Findings of the Association for Computational Linguistics: ACL 2025.
  70. Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025. Commonsense reasoning in Arab culture. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7695– 7710, Vienna, Austria. Association for Computational Linguistics. ↗
  71. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  72. Sumuk Shashidhar, Clémentine Fourrier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, and Dilek Hakkani-Tür. 2025. Yourbench: Easy custom evaluation sets for everyone. Preprint, arXiv:2504.01833. ↗
  73. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. 2025a. Fanar: An arabic-centric multimodal generative ai platform. Preprint, arXiv:2501.13944. ↗
  74. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025b. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  75. TII. 2025. Falcon-arabic: A breakthrough in arabic language models. find
  76. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, and 21 others. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. Preprint, arXiv:2204.07705. ↗
  77. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alzubaidi-2025-evaluating-arabic-large-language-models/