Elamur Research⌕
☰
home / papers / 2026 / QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

Paper · 2026 · Rawan El Ghali; Umm Kulsoom; Anas Madkoor; Dima Faris Alsaudi; Roaa Abdelmagid; Roaa Ibrahim; Raghad Mousa; Hamza Aljaji; Abdullah Khanafer; Abdallah Alkanani; Salah Feras Alali; Rawan Khaled Mohamed; and Ehsaneddin Asgari

Summary

QuranicMMLU is a benchmark that tests how well generative models handle the language of the Quran. Earlier Quranic and Islamic benchmarks mostly measure factual QA or retrieval, are usually multiple-choice only, and ignore cognitive difficulty.

The authors built a taxonomy of linguistic competence with five pillars (Phonology, Morphology, Syntax, Semantics, Pragmatics), 12 categories and 31 leaves. They then generated questions per leaf with Claude Opus 4.7, grounding them in sources such as the Quranic Arabic Corpus, its dependency treebank, Quran.com, tafsīr data and an asbāb al-nuzūl dataset. Items were spread across five Bloom levels and across verse-perplexity bands. GPT-5.2 and Gemini-2.5-Flash pre-annotated every item, and two human reviewers checked each one, leaving 980 items.

Every item exists in both open-ended and multiple-choice form, and 12 open systems were evaluated on both. Multiple-choice accuracy averages about 84%, while open-ended quality averages about 60%. The two rankings mostly agree, but some failures appear only once the answer choices are removed. The work matters because multiple-choice scores alone overstate model competence in a domain where errors are costly.

Key points

  • The taxonomy has 5 pillars, 12 categories and 31 leaves. It ranges from tajwīd phenomena (madd, qalqalah, waqf, qirāʾāt) through wazn, clitics, iḍāfa, ellipsis and disputed iʿrāb up to speech acts and inter-surah coherence.
  • Of 1,086 generated candidates, 610 were accepted unchanged and 370 were revised (235 of these only had their Bloom level corrected). A further 106 were dropped, giving 980 final items. Disagreements went to a majority vote of three adjudicators, and 133 items needed group discussion.
  • On the multiple-choice key, the generator and the two judge models agreed almost perfectly (Fleiss's κ = 0.90).
  • Ansari, the Islamic-specialized model, ranked first: 96.4% on multiple choice and 93.2% open-ended. ALLaM-7B was weakest on multiple choice (69.5%), and Qwen3-VL-4B was weakest open-ended (36.0%).
  • The two rankings correlate at Kendall's τ ≈ 0.73, with 57 of 66 model pairs in the same order. Fanar-Sadiq rose from 8th to 4th in open-ended scoring because other systems lost more ground.
  • Phonology was the hardest pillar, with most systems under 34% open-ended. Scores peaked at the Understand level and were lowest at Apply and Analyze.

Interesting ideas

  • I1. Each item is written once and asked both with and without answer options, so the gap between formats and any rank shifts expose ability that the options were masking.
  • I2. Test items are balanced across low, medium and high perplexity of the source text, as a rough measure of how familiar a model already is with it, so the score is not driven by memorized passages.
  • I3. Results are reported as a grid of narrow phenomena, each tied to an annotated corpus such as the dependency treebank, rather than as one aggregate number.
  • I4. Human reviewers follow a written ranking of what counts as truth: corpus evidence first, then scholarly consensus, then open dispute. Any item still disputed or suspected of hallucination is discarded rather than patched.
  • I5. Multiple-choice options follow a fixed recipe of one correct answer, two near-miss distractors and one clearly wrong option, which separates a model that can rule out nonsense from one that can make fine distinctions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Give the school-course tests and fable QA in both an options form and a free-answer form, and treat a large gap as a sign of guessing rather than real reading.
  • I2 (learn): Bin evaluation sentences by language-model perplexity as an extra negative control, to check that gains hold on unfamiliar text and not only on well-known passages.
  • I3 (tree): Break parser LAS down by construction (ellipsis, fronting, negation scope, possessives) so that a regression in one construction cannot hide inside a stable average.
  • I4 (learn): Adopt a similar written order of authority when adjudicating silver data from witness parsers, with gold treebank first, then teacher-tier agreement, and discard unresolved cases instead of forcing a label.
  • I5 (answer): Build generated quiz items for the readers on the same recipe, using near-miss distractors that differ only in a semantic role or an event order taken from the world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Hamad bin Khalifa UniversityDoha · Qatar71 works · 12,942 citations🎓American University of BeirutBeirut · Lebanon17 works · 1,847 citations🎓Carnegie Mellon University QatarDoha · Qatar13 works · 1,947 citations🎓University of BirminghamBirmingham · United Kingdom11 works · 62,331 citations🎓Qatar UniversityDoha · Qatar10 works · 81 citations

Also on the byline: University of Doha for Science and Technology.

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Education · 112 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Semitic languages · 1,484 works

Cite

Cite as Rawan El Ghali; Umm Kulsoom; Anas Madkoor; Dima Faris Alsaudi; Roaa Abdelmagid; Roaa Ibrahim; Raghad Mousa; Hamza Aljaji; Abdullah Khanafer; Abdallah Alkanani; Salah Feras Alali; Rawan Khaled Mohamed; and Ehsaneddin Asgari. 2026. QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge. arXiv:2609.22038 · Arabic NLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.22038 · arXiv 2609.22038

@misc{mohamed2026quranicmmlu,
  title = {{QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge}},
  author = {Rawan {El Ghali; Umm Kulsoom; Anas Madkoor; Dima Faris Alsaudi; Roaa Abdelmagid; Roaa Ibrahim; Raghad Mousa; Hamza Aljaji; Abdullah Khanafer; Abdallah Alkanani; Salah Feras Alali; Rawan Khaled Mohamed;} and Ehsaneddin Asgari},
  year = {2026},
  doi = {10.48550/arxiv.2609.22038},
  eprint = {2609.22038},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.22038}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, and Ehsaneddin Asgari. 2026. Reasoning over non-parametric memory in multilingual LLMs: A cognitive-based analysis of relational knowledge. In Third Conference on Language Modeling. find
  2. Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi, and Walid Magdy. 2026. Islamicmmlu: A benchmark for evaluating llms on islamic knowledge. Preprint, arXiv:2603.23750. ↗
  3. Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, and Ehsaneddin Asgari. 2026. Almieyar-oryx-bloombench: A bilingual multimodal benchmark for cognitively informed evaluation of vision-language models. In Findings of the Association for Computational Linguistics: ACL 2026. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele (a) MCQ accuracy (%) 96 88 87 87 84 83 74 81 68 68 61 65 98 97 96 96 95 96 90 87 86 78 82 83 95 93 87 89 85 82 79 72 67 67 59 63 97 96 97 95 94 93 92 81 85 82 79 73 93 95 93 89 95 89 95 80 77 68 69 59 96 93 92 91 90 88 84 80 76 73 70 69 Accuracy (%) 89 e yz 90 d n 84 l l era er ply Ap ate mb ta al alu ers Ov An me Ev d Un Re Bloom's levels Overall 92 87 74 82 75 74 56 74 51 56 44 53 97 89 67 81 64 48 57 67 37 49 32 46 91 78 50 70 52 41 48 61 34 40 30 45 95 75 54 68 53 39 48 63 33 51 28 39 86 92 74 82 72 70 66 70 66 61 64 49 93 83 62 76 62 52 53 67 41 50 36 46 Score (%) 54 e yz 61 d n 60 l l era er ply Ap ate mb ta al alu ers Ov An me Ev d Un Re Bloom's levels Overall Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics. ↗
  4. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. HAQA and QUQA: Constructing two Arabic question-answering corpora for the Quran and Hadith. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 90–97, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025. PALM: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics. find
  5. Lorin W. Anderson and David R. Krathwohl, editors. 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, complete edition. Longman, New York. find
  6. Farah Atif, Nursultan Askarbekuly, Kareem Darwish, and Monojit Choudhury. 2025. Sacred or synthetic? evaluating LLM reliability and abstention for religious questions. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(1):217–226. our card
  7. Adil Bahaj and Mounir Ghogho. 2025. MizanQA: Benchmarking large language models on Moroccan legal question answering. arXiv preprint arXiv:2508.16357. ↗ our card
  8. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631. ↗ our card
  9. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10308–10330, Bangkok, Thailand. Association for Computational Linguistics. find
  10. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390. ↗ our card
  11. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. ↗ our card
  12. Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz Al-Khatib, Logan Cochrane, Kareem Darwish, Rashid Yahiaoui, and Firoj Alam. 2026. From rag to agentic rag for faithful islamic question answering. Preprint, arXiv:2601.07528. ↗
  13. Abdessalam Bouchekif, Samer Rashwani, Emad Soliman Ali Mohamed, Mutaz Alkhatib, Heba Sbahi, Shahd Gaben, Wajdi Zaghouani, Aiman Erbad, and Mohammed Ghaly. 2025. QIAS 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 851–860, Suzhou, China. Association for Computational Linguistics. find
  14. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261. ↗ our card
  15. Kais Dukes, Eric Atwell, and Abdul-Baquee M. Sharaf. 2010. Syntactic annotation guidelines for the Quranic Arabic dependency treebank. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). our card
  16. Kais Dukes and Nizar Habash. 2010. Morphological annotation of Quranic Arabic. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). our card
  17. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Minhaj Ahmad, Abdulaziz Al-Homaid, Anas Al-Nuaimi, Enes Altinisik, Ehsaneddin Asgari, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Asim Ersoy, Masoomali Fatehkia, Mohammed Qusay Hashim, and 18 others. 2026. Fanar 2.0: Arabic generative ai stack. Preprint, arXiv:2603.16397. ↗
  18. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. 2025. Fanar: An arabic-centric multimodal generative ai platform. Preprint, arXiv:2501.13944. ↗ our card
  19. Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, and 304 others. 2026. Gemma 4 technical report. Preprint, arXiv:2607.02770. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  20. Spence Green and Christopher D. Manning. 2010. Better Arabic parsing: Baselines, evaluations, and analysis. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 394–402. our card
  21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR). ↗ our card
  22. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023. SALMA: Arabic sense-annotated corpus and WSD benchmarks. In Proceedings of ArabicNLP 2023, pages 359–369. find
  23. M Waleed Kadous, Amr Elsayed, Abdullah Al Nahas, and Ashraf Haress. 2026. Ansari: A retrieval-grounded islamic ai assistant – architecture, deployment, and lessons from 140,000 conversations. Preprint, arXiv:2608.20390. ↗
  24. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. ↗ our card
  25. David R. Krathwohl. 2002. A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41(4):212–218. ↗
  26. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing & Management, 59(6):103068. ↗
  27. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  28. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87, Marseille, France. European Language Resources Association. find
  29. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  30. Qur’an QA 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy Qur’an. In Proceedings of ArabicNLP 2023, pages 690–701, Singapore (Hybrid). Association for Computational Linguistics. find
  31. Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Mohamed Darwish, and Walid Magdy. 2025. IslamicEval 2025: The first shared task of capturing LLMs hallucination in islamic content. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 480–493, Suzhou, China. Association for Computational Linguistics. find
  32. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–110. our card
  33. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912. find
  34. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–374. ↗ our card
  35. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 248–258. our card
  36. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗ our card
  37. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of Arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 577–584, Sydney, Australia. Association for Computational Linguistics. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghali-2026-quranicmmlu-cognitively-aware-benchmark-evaluating/