Elamur Research⌕
☰
home / papers / 2025 / Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases

Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases

Summary

Dividing an estate under Islamic inheritance law (Ilm al-Mawārīth) means applying a dense set of fixed rules: deciding who inherits, who is excluded, and what fraction each heir gets. Done by hand, this work is slow and error-prone. The authors measure how well current LLMs handle this task using the Arabic QIAS 2025 shared-task data. The data are multiple-choice inheritance cases at three difficulty levels, built from IslamWeb fatwas and checked by an expert.

They compare three groups of systems:

  • open Arabic models (Falcon3, Fanar, Allam, ALLaM-Thinking),
  • proprietary models used through their APIs (Gemini Flash 2.5, Gemini Pro 2.5, GPT-4o, GPT o3),
  • fine-tuned versions of GPT-4o, Gemini Flash 2/2.5 and Llama 4 Scout.

They also test how sensitive the models are to a plain prompt versus a chain-of-thought prompt. The open Arabic models score poorly, below 40%. The top proprietary reasoning models reach about 92%. The strongest result, 92.7%, came from letting GPT o3, Gemini Pro 2.5 and Gemini Flash 2.5 vote by majority, and this ensemble placed third in the challenge. Fine-tuning helped GPT-4o but hurt Gemini. The study shows that rule-heavy legal reasoning still separates strong models from weak ones.

Key points

  • Dataset: QIAS 2025, six-option MCQs (A–F) at beginner, intermediate and advanced levels, sourced from a corpus of 32,000 IslamWeb fatwas.

    • Official split: 9,446 train / 1,000 validation / 1,000 unlabeled test.
    • For fine-tuning the authors used 7,000 train and 2,446 validation examples, and reported all comparisons on the separate 1,000-item validation set.

  • Base model accuracy:

    ModelAccuracy
    Falcon324.2%
    Fanar31.7%
    Allam38.8%
    ALLaM-Thinking29.2% (worse than plain Allam)
    GPT-4o70.1%
    Gemini Flash 2.591.5%
    GPT o392.3%

  • Prompt sensitivity: the chain-of-thought prompt raised GPT-4o from 57.5% to 70.1%. Gemini Flash 2.5 barely changed (90.7% vs 91.5%).

  • Fine-tuning:

    • LoRA on Llama 4 Scout (4-bit, 7 epochs) reached 84.3% with Prompt 1.
    • GPT-4o reached 84.7% without a system prompt and 86.6% with the CoT system prompt.
    • Gemini Flash 2.5 fell from 91.5% to 74.6%, and Gemini Flash 2 also degraded.
    • The tuned Llama models are published on Hugging Face as NYUAD_Llama4_Inheritance_Solver.

  • Best system: a three-model majority vote reached up to 92.7% and ranked third in the QIAS challenge.

Interesting ideas

  • I1. Combining a few strong but different models by simple majority vote beat every single model at every difficulty level, so cheap agreement among diverse systems pays off. — For Mova (learn): Supports voting across heterogeneous witness parsers and LLM teacher tiers for silver data, and suggests mixing model families rather than sizes of one family.
  • I2. Each model was scored under both a bare "answer with a letter" prompt and a step-by-step prompt, separating stable knowledge from prompt-induced gains. — For Mova (learn): When an LLM teacher labels frames or answers, run it under two prompt styles and treat labels that flip between them as low-confidence.
  • I3. The authors suspect that tuning a reasoning-optimized model on final labels alone erased its reasoning habits, since the supervision showed outputs but not derivations. — For Mova (learn): Train students on the intermediate steps (tree, frames, world state) and not only the final answer, so explainable structure is reinforced rather than shortcut.
  • I4. They note that a small accuracy gap from one run is not a real ranking, and propose repeated runs with mean and variance. — For Mova (learn): Report multiple seeds and run-to-run variance alongside negative controls before claiming one parser or reader beats another.
  • I5. Inheritance is a closed rule system over family relations and fractional holdings, which makes it a clean test of compositional reasoning with checkable answers. — For Mova (world): Kinship relations plus share arithmetic could serve as a benchmark for the world model's handling of entities, relations, holdings and measures, with deterministic gold answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.15796 · arXiv 2508.15796

@misc{aldahoul2025benchmarking,
  title = {{Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases}},
  author = {AlDahoul, Nouar and Zaki, Yasir},
  year = {2025},
  doi = {10.48550/arxiv.2508.15796},
  eprint = {2508.15796},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.15796}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The limitation of this work is that LLMs often lack comprehensive knowledge of all inheritance scenarios, which leads even the best reasoning models to occasionally select incorrect answers. Fine-tuning can help address this issue. However, a new problem arises: in the original dataset, answer choices were labeled without detailed reasoning for each option, which complicated the fine-tuning of reasoning-based models. To solve this, a second version of the dataset was released recently, including the reasoning behind each selected answer (Bouchekif et al., 2025; Bouchekif et al.). find
  2. Zubair Abbasi. 2025. Augmented learning: Generative artificial intelligence and islamic inheritance law. Accessed July 13, 2025. find
  3. Nouar AlDahoul, Talal Rahwan, and Yasir Zaki. 2024a. Polytc: a novel bert-based classifier to detect political leaning of youtube videos based on their titles. Journal of Big Data, 11(1):80. find
  4. Nouar AlDahoul, Myles Joshua Toledo Tan, Har-ishwar Reddy Kasireddy, and Yasir Zaki. 2024b. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123. ↗
  5. ALLaM-AI. 2025. Allam-7b-instruct-preview. https://huggingface.co/ALLaM-AI/ ALLaM-7B-Instruct-preview. Accessed: 2025-07-13. ↗
  6. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗ our card
  7. Abdessalam Bouchekif, Samer Rashwani, Emad Mohamed, Mutez Al-Khatib, Heba Sbahi, Shahd Gaben, Wajdi Zaghouani, Aiman Erbad, and Mohammed Ghaly. 2025. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutez Al-Khatib, and Mohammed Ghaly. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. CoRR. find
  8. Microsoft Copilot. 2025. Copilot ai companion. https://copilot.microsoft.com. Accessed: July 13, 2025. ↗
  9. DeepSeek. 2024. Deepseek chat. https://www.deepseek.com. Accessed: July 13, 2025. ↗
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  11. RG Jimoh, IR Adebayo, AO Ameen, and HO Ahmad. 2014. Design of an expert system to calculate inheritance shares based on islamic law. IEEE-African Journal of Computing & ICTs, 7(2):151–156. find
  12. Koray Kavukcuoglu. 2025. Gemini 2.5: Our most intelligent ai model. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking. ↗
  13. Chen Wei Kuo, Kevin Chu, Nouar AlDahoul, Hazem Ibrahim, Talal Rahwan, and Yasir Zaki. 2025. Neutralizing the narrative: Ai-powered debiasing of online news articles. arXiv preprint arXiv:2504.03520. ↗
  14. Meta AI. 2025a. The llama 4 herd: The beginning of a new era of natively multimodal ai. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/. Accessed: 2025-07-13; announces Scout & Maverick (17B active, MoE, 10 M & 1 M context) with native text+image support :contentRefer-ence[oaicite:2]index=2. ↗
  15. Meta AI. 2025b. Llama-4-scout-17b-16e-instruct. https://huggingface.co/meta-llama/ Llama-4-Scout-17B-16E-Instruct. Released April 5, 2025 under the Llama 4 Community License; multimodal MoE model with 17 B active parameters and 16 experts, 10M token context, supports 12 languages. ↗
  16. Marryam Yahya Mohammed, Sama Ayman Ali, Salma Khaled Ali, Ayad Abdul Majeed, and Ensaf Hussein Mohamed. 2025. Aftina: enhancing stability and preventing hallucination in ai-based islamic fatwa generation using llms and rag. Neural Computing and Applications, pages 1–26. find
  17. OpenAI. 2025. Introducing o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: 2025-07-13. ↗
  18. S Patel, H Kane, and R Patel. 2023. Building domain-specific llms faithful to the islamic worldview: Mirage or technical possibility?. arxiv. find
  19. QCRI. 2025. Fanar-1-9b-instruct. https://huggingface.co/QCRI/Fanar-1-9B-Instruct. Accessed: 2025-07-13. ↗
  20. Mohammed Al-Maghrabi Research. 2025. Allam-thinking: Arabic large language model with enhanced reasoning capabilities. https://huggingface.co/ almaghrabima/ALLaM-Thinking. ↗
  21. Research Center for Islamic Legislation and Ethics (CILE). 2025. Transforming islamic studies in the age of generative artificial intelligence (ai): Developing ai-empowered “islamgpt 1.0” for arabic sources. https://www.cilecenter.org/ research-publications/funded-projects/ transforming-islamic-studies-age-generative-artificial. ↗
  22. Islamic Will & Trust Services. 2025. Islamic law of inheritance according to quran and sunnah. https://islamicwillstrust.com/ islamic-law-of-inheritance/. Accessed July 2025. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. Fanar: An arabic-centric multimodal generative ai platform. ↗ our card
  23. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  24. TII Team. 2024. The falcon 3 family of open models. find
  25. Technology Innovation Institute (TII) UAE. 2024. Falcon3-7b-instruct. https://huggingface.co/ tiiuae/Falcon3-7B-Instruct. ↗
  26. Samia Zouaoui and Khaled Rezeg. 2021. Islamic inheritance calculation system based on arabic ontology (arafamonto). Journal of King Saud University-Computer and Information Sciences, 33(1):68–76. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldahoul-2025-benchmarking-legal-reasoning-llms-arabic/