Elamur Research⌕
☰
home / papers / 2025 / NYUAD at QIAS Shared Task: Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases

NYUAD at QIAS Shared Task: Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases

Paper · 2025 · Nouar AlDahoul and Yasir Zaki

Summary

The paper asks how well current large language models handle Islamic inheritance law (ʿIlm al-Mawārīth). This rule-heavy domain requires identifying the rightful heirs and computing their fractional shares, and doing it by hand is slow and error-prone. The authors use the QIAS 2025 shared-task data: Arabic multiple-choice questions drawn from expert-checked IslamWeb fatwas, with three difficulty tiers. They compare four groups of systems:

  • open Arabic models (Falcon3, Fanar, Allam, ALLaM-Thinking);
  • proprietary models used through their APIs (Gemini Flash/Pro 2.5, GPT-4o, GPT o3);
  • the same models under a plain prompt and a zero-shot chain-of-thought prompt;
  • supervised fine-tunes of Llama 4 Scout (LoRA), GPT-4o and Gemini Flash.

The open Arabic models score well under 40%, while the frontier reasoning models exceed 90% on validation. Fine-tuning helps GPT-4o but hurts Gemini. The final submission takes a majority vote over three strong base models and reaches 92.7% test accuracy, which placed third in the challenge. The study shows where LLMs stand on structured legal-arithmetic reasoning in Arabic.

Key points

  • Data. QIAS 2025 has six options (A–F) per question and beginner, intermediate and advanced levels. It is sourced from about 32,000 fatwas and split 9,446 / 1,000 / 1,000 into train, validation and test.

  • Base models on validation:

    ModelAccuracy
    Falcon324.2%
    Fanar31.7%
    Allam38.8%
    ALLaM-Thinking29.2%
    GPT-4o70.1%
    Gemini Flash 2.591.5%
    GPT o392.3%

  • Prompt sensitivity. Switching to the step-by-step prompt lifts GPT-4o from 57.5% to 70.1%. Gemini Flash 2.5 barely changes (90.7% vs 91.5%).

  • Fine-tuning:

    • GPT-4o rises to 84.7% without a system prompt and 86.6% with the CoT system prompt.
    • Llama 4 Scout (LoRA, 4-bit, 7 epochs) reaches 84.3% / 82.4%.
    • Gemini Flash 2.5 falls to 74.6%, and Flash 2 (adapter size 8) reaches only 64.6%.

  • Test set. Single models score o3 88.4%, Flash 2.5 88.1% and Pro 2.5 87.9%. A majority vote of the three gives 92.7%. The Llama fine-tunes are published as NYUAD_Llama4_Inheritance_Solver (two versions).

Interesting ideas

  • I1. A plain majority vote over three near-equal strong models gained about four points over the best single model, so their errors were largely uncorrelated.
  • I2. Fine-tuning a reasoning-optimized model on bare answer labels appeared to erode its step-by-step behaviour, and the authors suggest supervision needs the rationale, not just the final letter.
  • I3. Running every model under two prompt formats exposed which models really understand the task and which depend on prompt wording.
  • I4. The authors admit that a sub-point gap between two models from a single run is not meaningful and plan to report mean and variance over repeated runs.
  • I5. Inheritance questions combine identifying who exists in a family with exact fraction arithmetic over those people, which is structured world reasoning rather than text matching.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver trees could be accepted only where several witness parsers agree, with disagreements routed to an LLM teacher tier instead of being averaged away.
  • I2 (learn): When an LLM teacher labels data, it should store the justification chain together with the label, so trained components keep an explainable path.
  • I3 (learn): Evaluation could rephrase each test item in several surface forms and flag a component whose accuracy swings, as a robustness check next to negative controls.
  • I4 (learn): Parser or reader comparisons should report spread over several seeds or runs before declaring a winner.
  • I5 (answer): Such cases would make a good stress test for maths word-problem readers that must build entities and relations before computing shares.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 worksLegal AI · 134 works

Cite

Cite as Nouar AlDahoul and Yasir Zaki. 2025. NYUAD at QIAS Shared Task: Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.118

@inproceedings{aldahoul2025nyuad,
  title = {{NYUAD at QIAS Shared Task: Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases}},
  author = {Nouar AlDahoul and Yasir Zaki},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.118},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.118}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nouar AlDahoul, Talal Rahwan, and Yasir Zaki. 2024a. Polytc: a novel bert-based classifier to detect political leaning of youtube videos based on their titles. Journal of Big Data, 11(1):80. ↗
  2. Nouar AlDahoul, Myles Joshua Toledo Tan, Har-ishwar Reddy Kasireddy, and Yasir Zaki. 2024b. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123. assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics. ↗
  3. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
  4. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  5. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗
  6. RG Jimoh, IR Adebayo, AO Ameen, and HO Ahmad. 2014. Design of an expert system to calculate inheritance shares based on islamic law. IEEE-African Journal of Computing & ICTs, 7(2):151–156. find
  7. Chen Wei Kuo, Kevin Chu, Nouar AlDahoul, Hazem Ibrahim, Talal Rahwan, and Yasir Zaki. 2025. Neutralizing the narrative: Ai-powered debiasing of online news articles. arXiv preprint arXiv:2504.03520. ↗
  8. Marryam Yahya Mohammed, Sama Ayman Ali, Salma Khaled Ali, Ayad Abdul Majeed, and Ensaf Hussein Mohamed. 2025. Aftina: enhancing stability and preventing hallucination in ai-based islamic fatwa generation using llms and rag. Neural Computing and Applications, pages 1–26. ↗
  9. Shabaz Patel, Hassan Kane, and Rayhan Patel. 2023. Building domain-specific llms faithful to the islamic worldview: Mirage or technical possibility? arXiv preprint arXiv:2312.06652. ↗
  10. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867. ↗
  11. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. ↗
  12. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗
  13. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  14. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge Samia Zouaoui and Khaled Rezeg. 2021. Islamic inheritance calculation system based on arabic ontology (arafamonto). Journal of King Saud University-Computer and Information Sciences, 33(1):68–76. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldahoul-2025-nyuad-qias-shared-task-benchmarking/