Elamur Research⌕
☰
home / papers / 2026 / ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning

ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning

Paper · 2026 · Hussein S. Al-Olimat and Ahmad Alshareef

Summary

ALPS (Arabic Linguistic & Pragmatic Suite) addresses a gap in Arabic LLM evaluation. Most recent benchmarks favour size and often rely on translated or synthetic items, which leaves deep semantics and pragmatics (speech acts, implicature, presupposition, metaphor) under-tested. The authors built a small, native, expert-written challenge set of 531 multiple-choice questions covering 15 tasks and 47 subtasks. Items draw on Classical Arabic, MSA, Quranic text and poetry, and many hinge on i'rab (case marking). Distractors were made plausible on purpose, and every item passed at least two expert review rounds. Four trained annotators answered closed-book in a single sitting, and the gold answers were then settled by expert adjudication against reference works. The authors tested 23 models zero-shot. The strongest commercial system beats the average human, while Arabic-native models lag behind. Models are fluent but fail noticeably on dependencies that hinge on diacritics. The work shows that high benchmark scores can hide grammatical blind spots that may invert meaning.

Key points

  • The dataset has 531 items, 15 tasks and 47 subtasks, in two branches: Semantics and Pragmatics. It will be released under CC-BY-4.0 with a BIG-bench canary string to guard against training contamination.
  • Human baseline: four annotators averaged 84.6%, ranging from 74.2% to 90.8%. Oracle accuracy, meaning at least one annotator was right, reached 99.2%. Raw pairwise agreement was 79.8%, but Fleiss' κ was only 0.23.
  • By how many annotators answered correctly, items fall into tiers: 62.3% Easy (4/4), 20.7% Medium (3/4), 10.5% Hard (2/4) and 6.4% Challenging (0–1/4).
  • Human accuracy by task ranged from 73.8% on Speech Act Theory to 93.8% on Compositional Semantics.
  • Model results:
    • Gemini-3-flash scored highest at 94.2%.
    • The best Arabic-native model was Jais-2-70B at 83.6%, though it reached 95% on Implicature.
    • The lowest was NileChat-3B at 57.1%.
    • Error rate on diacritics-reliant tasks was 36.5%.
    • Evaluation was zero-shot, temperature 0, with no thinking modes or RAG.

Interesting ideas

  • I1. The authors grade each item's difficulty by how many independent annotators solved it, so a single human pass also produces a difficulty label for every item. This lets you report scores per difficulty tier instead of as one opaque number.
  • I2. They report oracle accuracy (did anyone get it right?) next to average accuracy. This separates broken or ambiguous items from items that are simply subtle, a cheap validity check for any hand-made test set.
  • I3. Distractors are built to look right on the surface but fail once the full context is read, so keyword matching cannot reach the answer.
  • I4. Minimal-pair items differ by one grammatical marker that flips agent and patient or turns a prohibition into a permission. They show directly whether a system's meaning depends on grammar or on fluency.
  • I5. Gold answers are settled by an expert who weighs annotators' arguments against references, not by majority vote, because the minority can be right on subtle items.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Gold test items could be tagged by how many witness parsers or human checkers got them right, so evaluation shows whether gains come only from easy items.
  • I2 (answer): School-test and fable QA sets could be audited this way: an item that no reader or human solves gets flagged for review rather than counted as a model failure.
  • I3 (answer): Multiple-choice options for fable QA could share words with the story while contradicting its event order or holdings, forcing the use of the world model rather than overlap.
  • I4 (frames): English minimal pairs that differ by one preposition, voice or attachment could test whether VerbNet role assignment flips correctly in the frames.
  • I5 (learn): When silver-data witness parsers disagree, an adjudication step based on explicit rules or a teacher tier, with recorded reasons, could replace plain voting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 worksImplicature · 138 works

Cite

Cite as Hussein S. Al-Olimat and Ahmad Alshareef. 2026. ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning. arXiv:2602.17054.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2602.17054 · arXiv 2602.17054

@misc{alolimat2026alps,
  title = {{ALPS: A Diagnostic Challenge Set for Arabic Linguistic \& Pragmatic Reasoning}},
  author = {Hussein S. Al-Olimat and Ahmad Alshareef},
  year = {2026},
  doi = {10.48550/arxiv.2602.17054},
  eprint = {2602.17054},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.17054}
}

References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin. 2025. Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations. Preprint, arXiv:2503.07833. ↗
  2. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  3. Khālid Khalīl Al-Huwaydī and Ni’mah Dahash Al-Tā’ī. 2015. Muḥāḍarāt fī al-Lisāniyyāt. Dār al-Kutub wa al-Wathā’iq, Baghdad. find
  4. Jalāl al-Dīn Al-Suyūṭī. 2003. Al-Alghāz al-Naḥwiyya: Al-Ṭirāz fī al-Alghāz, 1st edition. Al-Maktabah Al-Azhariyya lil-Turāth, Cairo. find
  5. Abū Manṣūr ’Abd al-Malik al Tha’alibī. 2009. Fiqh al-Lugha wa Asrār al-’Arabiyya. Al-Maktabah Al-Asriyyah, Cairo. find
  6. Aisha Alansari and Hamzah Luqman. 2025. Ara-HalluEval: A fine-grained hallucination evaluation framework for Arabic LLMs. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 148–161, Suzhou, China. Association for Computational Linguistics. find
  7. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. AraMed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 50–56, Torino, Italia. ELRA and ICCL.
  8. Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2025. AraTrust: An evaluation of trustworthiness for LLMs in Arabic. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8664–8679, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  9. Rawan Nasser Almatham, Kareem Mohamed Darwish, Raghad Al-Rasheed, Waad Thuwaini Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah A. Alzahrani, Eman Albilali, Nizar Habash, Abdelrahman Mustafa El-Sheikh, Muhammad Elmallah, Hamdy Mubarak, Zaid Alyafeai, Mohamed Anwar, Haonan Li, and 24 others. 2025. BALSAM: A platform for benchmarking Arabic large language models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, Suzhou, China. Association for Computational Linguistics.
  10. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics.
  11. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025. Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics.
  12. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of Arabic multimodal large language models and benchmarks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12753–12776, Bangkok, Thailand. Association for Computational Linguistics. ↗
  13. Ahmed Alzubaidi, Shaikha Alsuwaidi, Basma El Amel Boussaha, Leen AlQadi, Omar Alkaabi, Mohammed Alyafeai, Hamza Alobeidli, and Hakim Hacid. 2025. Evaluating arabic large language models: A survey of benchmarks, methods, and gaps. Preprint, arXiv:2510.13430. ↗
  14. Ali El Filali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open arabic llm leaderboard. https://huggingface.co/spaces/OALL/ Open-Arabic-LLM-Leaderboard. ↗
  15. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A challenging benchmark for Arabic language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.
  16. Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Husain Salem Abdulla Alharthi, Ines Riahi, Abduljalil Radman, Jorma Laaksonen, Fahad Shahbaz Khan, Salman Khan, and Rao Muhammad Anwer. 2025. CAMEL-bench: A comprehensive Arabic LMM benchmark. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1970–1980, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  17. Cristina Grisot. 2018. Corpus-Based Contrastive Study of Verbal Tenses, pages 111–135. Springer International Publishing, Cham. find
  18. Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah Al-Shamlan, Omar Knio, and George Turkiyyah. 2024. ArabLegalEval: A multitask benchmark for assessing Arabic legal knowledge in large language models. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 225–249, Bangkok, Thailand. Association for Computational Linguistics.
  19. Muwaffaq al-Dīn Ya’īsh ibn ’Alī Ibn Ya’īsh. n.d. Sharḥ al-Mufaṣṣal li-l-Zamakhsharī. Idārat al-Ṭibā’ah al-Munīriyyah, Cairo. Classic Muniriyya edition (reprinted ca. 1928 or later). find
  20. Muḥammad Khaṭṭābī. 1991. Lisāniyyāt al-Naṣṣ: Mad-khal ilā Insijām al-Khiṭāb, 1st edition. Al-Markaz al-Thaqāfī al-’Arabī, Beirut/Casablanca. find
  21. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. ↗
  22. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, page 552–561. AAAI Press. find
  23. Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Taher Abdelfadil, Shady Shehata, and Muhammad Abdul-Mageed. 2025. JAWAHER: A multidialectal dataset of Arabic proverbs for LLM benchmarking. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12320–12341, Albuquerque, New Mexico. Association for Computational Linguistics. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. ↗
  24. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.
  25. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. Preprint, arXiv:2501.00559. ↗
  26. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023. Dolphin: A challenging and diverse benchmark for Arabic NLG. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1404–1422, Singapore. Association for Computational Linguistics.
  27. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics. ↗
  28. Mas’ūd Saḥrāwī. 2005. Al-Tadāwuliyya ’inda al-’Ulamā’ al-’Arab: Dirāsa Tadāwuliyya li-Zāhirat al-Af’āl al-Kalāmiyya fī al-Turāth al-Lisānī al-’Arabī, 1st edition. Dār al-Ṭalī’ah, Beirut. find
  29. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-olimat-2026-alps-diagnostic-challenge-set-arabic/