Elamur Research⌕
☰
home / papers / 2025 / QU-NLP at QIAS 2025 Shared Task: A Two-Phase LLM Fine-Tuning and Retrieval-Augmented Generation Approach for Islamic Inheritance Reasoning

QU-NLP at QIAS 2025 Shared Task: A Two-Phase LLM Fine-Tuning and Retrieval-Augmented Generation Approach for Islamic Inheritance Reasoning

Paper · 2025 · Mohammad AL-Smadi

Summary

The paper describes QU-NLP's entry to SubTask 1 of QIAS 2025, which tests whether LLMs can reason about Islamic inheritance law (Ilm al-Mawārīth). Each item is a multiple-choice question with one correct option. Solving one means identifying which heirs qualify and which are excluded, applying fixed shares and residuary rules, handling special cases such as ʿawl and radd, and computing the final portions.

The author adapted the Arabic-focused Fanar-1-9B with LoRA on 20,000 training questions, using 4-bit NF4 quantization and gradient checkpointing to fit memory limits. At inference, a RAG layer retrieves supporting passages and passes them to the model.

On the 1,000-question test set the system reached 85.8% accuracy. That is below o3 and Gemini 2.5 but well above GPT-4.5 and the open models run zero-shot. On the advanced half it scored 97.6%, the best of any system compared.

The ablation shows that almost all of the gain comes from fine-tuning, not from retrieval. The error analysis traces most failures to blocked heirs, negation cues and dataset artifacts. The overall message is that a mid-sized, domain-tuned model can rival frontier models on a narrow, rule-heavy domain.

Key points

  • Data and model: The dataset has 22,000 MCQs (20k train, 1k validation, 1k test), each with up to six options (A–F) and split evenly into Beginner and Advanced levels.

    • The base model is Fanar-1-9B, released after tuning as Fanar-1-9B-Islamic-Inheritance-Reasoning.
    • LoRA adapters (r=32, α=64, dropout 0.1) cover all attention and MLP projections.

  • Retrieval and decoding: all-MiniLM-L6-v2 embeddings and a FAISS index supply the top-5 passages.

    • The model is trained to emit only the gold letter.
    • Decoding is greedy, with temperature 0.05 and at most 15 new tokens; a regex extracts the chosen letter.

  • Results:

    SystemOverallBeginnerAdvanced
    QU-NLP85.874.097.6
    o393.4––
    Gemini 2.590.6––
    GPT-4.574.0––

  • Ablation:

    • The base model scores 18.6.
    • Adding LoRA raises it to 86.5.
    • Adding RAG on top gives 85.8: slightly better on Advanced (96.8→97.6) and worse on Beginner (76.2→74.0).

  • Errors (142 total):

    • Questions whose answer is a blocked heir (maḥjūb) reach only 64.5%, against 94.9% for all other questions. Such items are 1.70% of dev, 17.46% of train and 29.90% of test.
    • Questions containing negation or exception cues reach 83.5%, against 95.3% for the rest.
    • 10 errors (about 7%) come from answer options that differ only in orthography.

Interesting ideas

  • I1. Reporting each component's effect per difficulty level exposed that retrieval helps hard items but adds noise to easy ones, which a single overall score would have hidden.
  • I2. The author compared how often one answer class appears in train, dev and test, and found the dev set nearly lacked the class that dominated test errors.
  • I3. Cutting the error analysis by surface cues such as negation and exception words revealed polarity flips, and the proposed fix is to augment training with rewritten negated variants.
  • I4. Normalizing orthography and deduplicating options before scoring separates real model errors from label artifacts caused by spelling variants.
  • I5. For heirs excluded by higher-priority relatives, the author suggests adding explicit post-rules on top of the neural model rather than relying on it alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Ablations of each pipeline stage could be reported per slice (easy/hard, short/long sentences), so a module that helps one slice and hurts another is visible.
  • I2 (learn): Before trusting dev scores, Mova's evaluation could compare phenomenon frequencies (negation, passives, coordination) across its splits and flag any that are badly under-represented.
  • I3 (frames): Negation and exception scope could be an explicit frame attribute, tested with paired positive/negated sentences as a contrastive check.
  • I4 (answer): School-test readers could normalize and merge near-identical options before choosing, and the evaluator could count such collapses separately rather than as wrong answers.
  • I5 (world): Priority and exclusion constraints (who blocks whom in a holding or entitlement) could be compiled as symbolic rules that check and override the world state, keeping each decision explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Qatar UniversityDoha · Qatar53 works · 1,027 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksSemitic languages · 3,066 worksInformation retrieval & search · 552 works

Cite

Cite as Mohammad AL-Smadi. 2025. QU-NLP at QIAS 2025 Shared Task: A Two-Phase LLM Fine-Tuning and Retrieval-Augmented Generation Approach for Islamic Inheritance Reasoning. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.123

@inproceedings{alsmadi2025qu,
  title = {{QU-NLP at QIAS 2025 Shared Task: A Two-Phase LLM Fine-Tuning and Retrieval-Augmented Generation Approach for Islamic Inheritance Reasoning}},
  author = {Mohammad AL-Smadi},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.123},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.123}
}

References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, and 22 others. 2025. Fanar: An arabic-centric multimodal generative ai platform.
  2. Alaa N. Akkila and Samy S. Abu Naser. 2016. Proposed expert system for calculating inheritance in islam. World Wide Journal of Multidisciplinary Research and Development, 2(9):38–48. find
  3. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydin. 2024. A rag-based question answering system proposal for understanding islam: Mufassirqas llm. arXiv preprint. find
  4. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, and 1 others. 2023. The falcon series of open language models. arXiv preprint.
  5. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Is gpt-4 a good islamic expert for answering quran questions? In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages 124–133. find
  6. M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Alrashed, Faisal Mirza, Shaykhah Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, and 1 others. 2024. Allam: Large language models for arabic and english. arXiv preprint.
  7. Abdessalam Bouchekif, Samer Rashwani, Emad Mohamed, Mutaz Al-Khatib, Heba Sbahi, Shahd Gaben, Wajdi Zaghouani, Aiman Erbad, and Mohammed Ghaly. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics.
  8. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
  9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc.
  10. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The faiss library.
  11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. ↗
  12. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547. find
  13. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459– 9474.
  14. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. Qur’an qa 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), pages 79–87, Marseille, France. European Language Resources Association.
  15. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. Qur’an qa 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an. In Proceedings of ArabicNLP 2023, pages 690–701, Singapore.
  16. Marryam Yahya Mohammed, Sama Ayman Ali, Salma Khaled Ali, Ayad Abdul Majeed, and Ensaf Hussein Mohamed. 2025. Aftina: Enhancing stability and preventing hallucination in ai-based islamic fatwa generation using llms and rag. Neural Computing and Applications, pages 1–26. find
  17. Omar T Mohammedi. 2012. Sharia-complaint wills; principles, recognition, and enforcement. NYL Sch. L. Rev., 57:259. find
  18. PyTorch Team. 2025. Gradient checkpointing for large models. https://pytorch.org/docs/stable/ checkpoint.html. ↗
  19. Mohammad Amaan Sayeed, Mohammed Talha Alam, Raza Imam, Shahab Saquib Sohail, and Amir Hussain. 2025. From rag to agentic: Validating islamic-medicine responses with llm agents. arXiv preprint. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Parameter Decoding Temperature Top-p Max new tokens Input length Repetition penalty Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint.
  20. Sadia Tabassum, A. H. M. Hoque, Sharaban Twahura, and Mohammad Osiur Rahman. 2019. Developing an islamic farayez system applying software engineering. Jurnal Kejuruteraan, 31(1):25–38. find
  21. Samia Zouaoui and Khaled Rezeg. 2021. Islamic inheritance calculation system based on arabic ontology (arafamonto). Journal of King Saud University-Computer and Information Sciences, 33(1):68–76. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/smadi-2025-qu-nlp-qias-2025-shared/