Elamur Research⌕
☰
home / papers / 2025 / From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation

From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation

Summary

The authors argue that current Arabic LLM benchmarks often fall short. Many are machine-translated or built from school materials. They can contain grammatical slips, inconsistent formatting and culturally foreign content, and they rarely test deep specialist knowledge. The paper responds in three steps. First, it sets out construction guidelines in four areas: linguistic standards, cultural alignment, methodological and structural standards, and evaluator requirements. Second, it applies these guidelines to audit Ghafa, ArabicMMLU and INCLUDE, cataloguing concrete defects. In one case, translation flipped a religious fact. Third, it releases the Arabic Depth Mini Dataset (ADMD): 490 hard, hand-written questions drawn from books, covering ten domains and 42 sub-domains. Five models answered ADMD: GPT-4, Claude 3.5 Sonnet, Gemini Flash 1.5, CommandR 100B and Qwen-Max. Humans graded every answer on a four-level scale instead of using an LLM judge. Claude 3.5 Sonnet came first, yet it answered only about 30% fully correctly. All models did poorly on Islamic studies and on linguistics and literature. The study shows that culturally grounded, expert-level Arabic questions remain largely unsolved and that benchmark quality needs closer scrutiny.

Key points

  • ADMD holds 490 questions in 10 domains and 42 sub-domains. Four in-house researchers (three Syrian, one Yemeni) wrote them from books and references. Only a sample is planned for public release.
  • Answers are graded on four levels:
    • True: fully correct.
    • Partially-True: roughly 60–80% correct.
    • Partially-False: roughly 20–30% correct.
    • False: incorrect.
  • Claude 3.5 Sonnet got 147 of 490 answers fully correct. It scored about 50% in Mathematics & Computational Sciences and in Philosophy & Logic, and 51.67% in General & Miscellaneous Sciences. GPT-4 was weakest, with 44 True and 355 False answers.
  • Error rates peaked in two domains, Linguistics & Literature and Islamic & Religious Studies. Claude's False rate there was 41.82%, against more than 80% for the other models.
  • A comparison table labels each Arabic benchmark by origin: reviewed, handwritten, generated or translated. ADMD is the only one marked reviewed and handwritten with no generated or translated material.

Interesting ideas

  • I1. Grading answers on a four-step partial-credit scale shows how wrong a model is, which a single right/wrong bit hides. Partial credit separates near-misses from complete failures. — For Mova (answer): Graded scoring for fable QA and word-problem answers would show whether an error came from one missing fact in the world or from a wrong frame overall.
  • I2. Each benchmark is labelled by provenance (handwritten, translated, generated, human-reviewed), so its reliability can be judged before its scores are trusted. — For Mova (learn): Tagging every silver and evaluation item with its origin would let us check whether LLM-teacher or converted data inflates scores compared with hand-written items.
  • I3. Before building anything new, the authors audit existing test sets item by item for translation artifacts and factual inversions, and they find benchmark errors that distort model scores. — For Mova (tree): The same kind of audit could run on converted treebanks and UD dialect conversions to find systematically wrong labels before they are used for training or scoring.
  • I4. Reporting results per domain shows that an overall average hides sharp differences, such as strong logic alongside weak culture-specific knowledge. — For Mova (answer): School-course tests should be scored per subject and per question type, so weak stages of the pipeline show up instead of being averaged away.
  • I5. The authors deliberately skip an LLM judge for non-English evaluation and use careful human review, accepting slower scaling in exchange for trustworthy grades. — For Mova (learn): A small, human-graded gold set should anchor any automatic or LLM-based evaluation, so drift in the automatic judge can be detected.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.01920 · arXiv 2506.01920

@misc{sibaee2025guidelines,
  title = {{From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation}},
  author = {Sibaee, Serry and Nacar, Omer and Ammar, Adel and Al-Habashi, Yasser and Al-Batati, Abdulrahman and Boulila, Wadii},
  year = {2025},
  doi = {10.48550/arxiv.2506.01920},
  eprint = {2506.01920},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.01920}
}

References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. Arabicaqa: A comprehensive dataset for arabic question answering. Preprint, arXiv:2403.17848. ↗ our card
  2. Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2024. Aratrust: An evaluation of trustworthiness for llms in arabic. Preprint, arXiv:2403.09017. ↗
  3. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics. find
  4. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2024. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390. ↗ our card
  5. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. ↗ our card
  6. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A challenging benchmark for Arabic language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics. find
  7. Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation. Preprint, arXiv:2502.06559. ↗
  8. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗ our card
  10. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 220–247, Singapore. Association for Computational Linguistics. find
  11. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. find our card
  12. KSAA. 2024. Balsam benchmark for evaluating arabic large language models (llms). https://benchmarks.ksaa.gov.sa/b/balsam. Balsam is a collaborative initiative between prominent academic and governmental institutions in the Middle East. It aims to lead the development and provisioning of specialized evaluation datasets essential for assessing the performance of Arabic Large Language Models (LLMs) across a wide range of Natural Language Processing (NLP) tasks. ↗
  13. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. ↗
  14. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. find
  15. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. Preprint, arXiv:2501.00559. ↗ our card
  16. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S. Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader, and Anis Koubaa. 2025. Towards inclusive Arabic LLMs: A culturally aligned benchmark in Arabic large language model evaluation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 387–401, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sibaee-2025-guidelines-practice-new-paradigm-arabic/