Elamur Research⌕
☰
home / papers / 2025 / Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation

Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation

Paper · 2025 · Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi et al.

Summary

The paper asks how well current LLMs handle Islamic inheritance law (ʿilm al-mawārīth). The task is a demanding mix of reading the family situation, applying exclusion and fixed-share rules, and doing exact fractional arithmetic with corrections such as radd and ʿawl. The authors test seven models zero-shot with Arabic prompts. The benchmark is the 1,000-question validation set of QIAS 2025 Shared Task 2. Its questions were produced from IslamWeb fatwas by Gemini 2.5 Pro and then checked by Islamic-studies experts. Each model must pick one of six options and explain its choice.

Reasoning-oriented models do far better than the rest. o3 and Gemini 2.5 exceed 90%, while the Arabic-focused and open models stay below 50% and degrade sharply on advanced cases. The authors also build an error taxonomy and audit the explanations behind correct answers. Weaker models often reach the right option through flawed logic or invented citations. The main lesson is that accuracy alone overstates legal competence, so evaluation in rule-governed domains must also examine the justification.

Key points

  • Benchmark: 1,000 MCQs, six options with one correct answer, split into 500 Beginner and 500 Advanced. The data come from QIAS 2025 Shared Task 2, and options were cleaned so that no two are equivalent (e.g. 1/2 vs 2/4).
  • Overall accuracy: o3 93.4%, Gemini 2.5 90.6%, GPT-4.5 74.0%, LLaMA 3 70B 48.8%, Fanar 48.1%, Mistral-Saba-24B 44.5%, ALLaM-7B 42.9%.
  • ALLaM falls from 58.0% on Beginner to 27.8% on Advanced. o3 and Gemini lose only about two points between levels.
  • Error analysis used 142 questions that every low-scoring model got wrong. Errors were grouped as foundational (comprehension, rule application) or complex, such as exceptional or disputed cases like munāsakha and the grandfather-with-siblings case.
  • In 40 manually checked correct answers, foundational flaws appeared in the justifications of 27.5% (ALLaM), 32.5% (Fanar) and 47% (LLaMA). Gemini's explanations were sound.

Interesting ideas

  • I1. The authors read the explanations attached to correct answers and found that a right choice can hide wrong reasoning, so they score answer and rationale separately.
  • I2. They removed answer options that are numerically or semantically equivalent, so each question has exactly one acceptable answer and grading stays clean.
  • I3. The hard error-analysis subset is defined as the items that all weak systems miss, which concentrates attention on systematic gaps rather than random noise.
  • I4. Splitting items into a foundational level and an advanced level separates missing basic knowledge from failure at multi-step composition.
  • I5. Inheritance solving is framed as a pipeline: identify relationships and conditions, determine eligible and excluded heirs, compute shares, then apply corrections like radd or ʿawl.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could flag items where the final answer matches but the explanation chain through tree, frames and world is wrong, treating them as hidden failures rather than successes.
  • I2 (answer): Test sets for maths word problems could normalise fractions and units before comparing options, so that equivalent forms are not scored as different answers.
  • I3 (learn): Questions failed by every witness parser or student tier could form a targeted pool for repair-rule induction and teacher escalation.
  • I4 (answer): School-course and fable QA tests could be tagged by required depth (single fact versus chained events or states), showing whether errors come from the knowledge base or from composition.
  • I5 (world): This is a good stress case for the world model, where holdings, kinship entities and ordered events (successive deaths) must be tracked explicitly before any arithmetic over measures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 worksLegal AI · 134 works

Cite

Cite as Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al Khatib and Mohammed Ghaly. 2025. Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.20

@inproceedings{bouchekif2025assessing,
  title = {{Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation}},
  author = {Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz {Al Khatib} and Mohammed Ghaly},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.20},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.20}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur A. Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed K. Elmagarmid, Mohamed Y. Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, and 22 others. 2025. Fanar: An arabic-centric multimodal generative AI platform. arXiv preprint, arXiv:2501.13944. ↗
  2. Anis Koubaa, Adel Ammar, Lahouari Ghouti, Omar Najar, and Serry Sibaee. 2024. Arabiangpt: Native arabic gpt-based large language model. arXiv preprint arXiv:2402.15313. ↗
  3. Alaa N Akkila and Samy S Abu Naser. 2016. Proposed expert system for calculating inheritance in islam. Technical report. find
  4. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  5. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 79–87, Marseille, France. European Language Resources Association.
  6. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydin. 2024. A rag-based question answering system proposal for understanding islam: Mufassirqas llm. arXiv preprint arXiv:2401.15378. ↗
  7. Hayfa A Aleid and Aqil M Azmi. 2025. Hajj-fqa: A benchmark arabic dataset for developing question-answering systems on hajj fatwas: H. aleid and a. azmi. Journal of King Saud University Computer and Information Sciences, 37(6):135. ↗
  8. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, and 1 others. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867. ↗
  9. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Is gpt-4 a good islamic expert for answering quran questions? In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages 124–133. find
  10. M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Alrashed, Faisal Mirza, Shaykhah Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗
  11. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Second Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics.
  12. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. Preprint, arXiv:2509.01081. ↗
  13. Marryam Yahya Mohammed, Sama Ayman Ali, Salma Khaled Ali, Ayad Abdul Majeed, and Ensaf Hussein Mohamed. 2025. Aftina: enhancing stability and preventing hallucination in ai-based islamic fatwa generation using llms and rag. Neural Computing and Applications, pages 1–26. ↗
  14. Faiza Qamar, Seemab Latif, and Rabia Latif. 2024. A benchmark dataset with larger context for non-factoid question answering over islamic text. arXiv preprint arXiv:2409.09844. ↗
  15. Syed Qamar and 1 others. 2023. A benchmark dataset with larger context for qa over islamic text. arXiv preprint. find
  16. Muhammad Razif Rizqullah, Ayu Purwarianti, and Alham Fikri Aji. 2023. Qasina: Religious domain question answering using sirah nabawiyah. In 2023 10th International Conference on Advanced Informatics: Concept, Theory and Application (ICAICTA), pages 1–6. find
  17. Mohammad Amaan Sayeed, Mohammed Talha Alam, Raza Imam, Shahab Saquib Sohail, and Amir Hussain. 2025. From rag to agentic: Validating islamic-medicine responses with llm agents. arXiv preprint arXiv:2506.15911. ↗
  18. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  19. Sadia Tabassum, AHM Hoque, Sharaban Twahura, and Mohammad Osiur Rahman. 2019. Developing an islamic farayez system applying software engineering. Jurnal Kejuruteraan, 31(1):25–38. ↗
  20. Samia Zouaoui and Khaled Rezeg. 2021. Islamic inheritance calculation system based on arabic ontology (arafamonto). Journal of King Saud University-Computer and Information Sciences, 33(1):68–76. find
  21. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, and 1 others. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bouchekif-2025-assessing-large-language-models-islamic/