Elamur Research⌕
☰
home / papers / 2026 / AraLingBench: A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

AraLingBench: A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

Paper · 2026 · Mohamad Bilal Zbib, Hasan Abed Al Kader Hammoud, Ammar Mohanna et al.

Summary

Arabic LLM evaluation has focused mostly on factual knowledge and problem solving, through suites such as ArabicMMLU, EXAMS, BALSAM and 3LM. These rarely check whether a model actually controls the structure of the language. AraLingBench addresses this with 150 multiple-choice items covering five areas: grammar (Nahw), morphology (Sarf), spelling (Imlaa), reading comprehension and syntax. Five Arabic linguistics experts at the American University of Beirut wrote the items. The items then went through difficulty and diversity filtering, senior-linguist quality control, and majority-vote difficulty labelling.

The authors tested 35 Arabic, bilingual and multilingual models in a zero-shot, single-letter-answer setup, with no few-shot examples or chain-of-thought. Models handled spelling and comprehension reasonably well but were weak on morphology and especially syntax. Scores track general benchmarks overall, yet some training regimes appear to raise knowledge-benchmark results without deeper linguistic skill. Retrieval-heavy and synthetic-data setups are examples. The benchmark therefore gives a diagnostic signal that knowledge-oriented tests miss, which can guide the development of Arabic LLMs.

Key points

  • Dataset design: 150 questions, 30 per category. By difficulty, 50 are Easy, 74 Medium and 26 Hard. 125 items have four options and 25 have three, and answer positions show no systematic bias. The data and code are on Hugging Face and GitHub.
  • Models tested: 35 models ranging from 350M to 70B parameters, including Hala, Fanar, Yehia, JAIS, ALLaM, Qwen2.5 and Phi-4. ALLaM-7B-Instruct-preview and Yehia-7B-preview top the weighted ranking.
  • Category gap: even the strongest models reach about 86.7% on Spelling but stall near 60% on Syntax.
  • Skill correlations: grammar and morphology are closely linked, while syntax behaves largely independently. Correlation with general benchmarks is high (r > 0.75), but it does not guarantee linguistic competence.
  • Difficulty labels: human-assigned difficulty does not map monotonically onto model accuracy, although model rankings stay consistent across levels (r > 0.65).
  • Stated limitations: the benchmark is small, uses only the MCQ format, and has category boundaries that can overlap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia27 works · 1,084 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksMorphology · 405 works

Cite

Cite as Mohamad Bilal Zbib, Hasan Abed Al Kader Hammoud, Ammar Mohanna, Nadine Rizk, Fatima Karnib, Sina Moukaled and Bernard Ghanem. 2026. AraLingBench: A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.45

@inproceedings{zbib2026aralingbench,
  title = {{AraLingBench: A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models}},
  author = {Mohamad Bilal Zbib, Hasan Abed {Al Kader Hammoud}, Ammar Mohanna, Nadine Rizk, Fatima Karnib, Sina Moukaled and Bernard Ghanem},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.45},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.45}
}

References

29 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In ACL, pages 7088–7105, Online. ACL. ↗
  2. Shahad Al-Khalifa, Nadir Durrani, Hend Al-Khalifa, and Firoj Alam. 2025. The landscape of arabic large language models. Commun. ACM, 68(10):54–61. ↗
  3. Rawan Nasser Almatham, Kareem Mohamed Darwish, Raghad Al-Rasheed, Waad Thuwaini Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, and 1 others. 2025. Balsam: A platform for benchmarking arabic large language models. In ArabicNLP, pages 258–277.
  4. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Arabic-NLP, pages 244–275, Singapore (Hybrid). ACL.
  5. Malik H. Altakrori, Nizar Habash, Abdelhakim Freihat, Younes Samih, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Teresa Lynn, Preslav Nakov, and Alham Fikri Aji. 2025. Dialectalarabicmmlu: Benchmarking dialectal capabilities in arabic and multilingual language models. arXiv preprint arXiv:2510.27543v1. ↗
  6. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025. Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In ACL, pages 32871–32894, Vienna, Austria. ACL.
  7. Ahmed Alzubaidi, Shaikha Alsuwaidi, Basma El Amel Boussaha, Leen AlQadi, Omar Alkaabi, Mohammed Alyafeai, Hamza Alobeidli, and Hakim Hacid. 2025. Evaluating arabic large language models: A survey of benchmarks, methods, and gaps. arXiv preprint arXiv:2510.13430v2. ↗
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In OSACT, pages 9–15, Marseille, France. European Language Resource Association. ↗
  9. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, and 7 others. 2025. AL-Lam: Large language models for arabic and english. In ICLR. find
  10. Basma El Amel Boussaha, Leen Al Qadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, and Hakim Hacid. 2025. 3LM: Bridging Arabic, STEM, and code through benchmarking. In ArabicNLP, pages 42–63, Suzhou, China. ACL. ↗
  11. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E. Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv preprint arXiv:2505.03427v2. ↗
  12. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A challenging benchmark for Arabic language understanding. In Findings of ACL, pages 9559–9586, Toronto, Canada. ACL.
  13. Fanar Team. 2025. Fanar: An arabic-centric multimodal generative ai platform. Preprint, arXiv:2501.13944. Authors: Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, Chaoyi Ruan. ↗
  14. Abbas Ghaddar, Yimeng Wu, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Duan Xinyu, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, and Philippe Langlais. 2021. Jaber: Junior arabic bert. ArXiv, abs/2112.04329. ↗
  15. Hasan Abed Al Kader Hammoud, Mohammad Zbeeb, and Bernard Ghanem. 2025. Hala technical report: Building arabic-centric instruction & translation models at scale. arXiv preprint arXiv:2509.14008v1. ↗
  16. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In EMNLP, pages 5427– 5444, Online. ACL. find
  17. Faris Hijazi, Somayah Alharbi, Abdulaziz AlHussein, Harethah Shairah, Reem Alzahrani, Hebah Al-shamlan, George Turkiyyah, and Omar Knio. 2024. ArabLegalEval: A multitask benchmark for assessing Arabic legal knowledge in large language models. In ArabicNLP, pages 225–249, Bangkok, Thailand. ACL.
  18. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  19. Inception. 2024. Jais family model card. find
  20. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  21. Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025. Commonsense reasoning in Arab culture. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7695– 7710, Vienna, Austria. Association for Computational Linguistics. find
  22. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149v2. ↗
  23. Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis, and Eric Xing. 2025. Atlas-chat: Adapting large language models for low-resource Moroccan Arabic dialect. In LoResLM, pages 9–30, Abu Dhabi, United Arab Emirates. ACL. ↗
  24. ZeroOne AI. 2025. Suhail-14b-preview. find
  25. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of ACL 2024, pages 5622–5640, Bangkok, Thailand. ACL. ↗
  26. Anis Koubaa, Adel Ammar, Lahouari Ghouti, Omar Najar, and Serry Sibaee. 2024. Arabiangpt: Native arabic gpt-based large language model. Preprint, arXiv:2402.15313. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. ↗
  27. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In COLING, pages 4186– 4218, Abu Dhabi, UAE. ACL.
  28. Navid-AI. 2025. Yehia 7b preview. find
  29. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. Preprint, arXiv:2409.12623. Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zbib-2026-aralingbench-human-annotated-benchmark-evaluating/