Elamur Research⌕
☰
home / papers / 2025 / Athar at QIAS2025: LLM-based Question Answering Systems for Islamic Inheritance and Classical Islamic Knowledge

Athar at QIAS2025: LLM-based Question Answering Systems for Islamic Inheritance and Classical Islamic Knowledge

Paper · 2025 · Yossra Noureldien, Hassan Suliman, Farah Attallah et al.

Summary

The paper describes the Athar team's entry to QIAS2025, a shared task that tests LLMs on Arabic multiple-choice questions about Islamic knowledge. It has two tracks. The first covers inheritance law (ʿilm al-mawārīth), which requires applying share rules, exclusion (ḥajb) and proportional reduction (ʿawl). The second draws on classical Islamic books.

For inheritance, the authors used no fine-tuning. They sent an Arabic expert-persona prompt to DeepSeek-R1 in zero-shot mode, forced the answer into a tagged label, and pulled that label out with a regex.

For the classical-texts track, they built a three-stage RAG system:

  • chunked the 23 books;
  • retrieved passages in parallel with dense FAISS search and BM25;
  • had GPT-4o-mini rerank the pooled candidates;
  • let GPT-4o pick the option.

The inheritance system scored 0.704 on test, placing 10th of 15 teams, after reaching 0.885 on dev. The RAG system scored 0.927, placing 2nd of 10.

The work matters because it shows a clear contrast. Retrieval grounding works well for factual recall from source texts. Long multi-heir calculations, however, still trip up a strong reasoning model when inputs grow longer.

Key points

  • SubTask 1 data: 20,000 train, 1,000 dev and 1,000 test questions, each with six options (A–F), plus 3,165 IslamWeb fatwas as extra material. SubTask 2 data: a 700-question dev set from 21 books and a 1,000-question test set from 23 books, two of which never appear in dev.
  • Retrieval setup: RecursiveCharacterTextSplitter (500-character chunks, 50 overlap), text-embedding-3-large vectors in FAISS IndexFlatIP, and Okapi BM25. The top 7 hits from each retriever form a pool of 14, GPT-4o-mini keeps 5, and GPT-4o answers.
  • Scores against the leaders: SubTask 1 test accuracy was 70.4% (Wilson CI 67.5–73.2%), against a leader at 0.972. SubTask 2 test accuracy was 0.927, against a best of 0.937.
  • SubTask 1 dev-to-test drop:
    • Test questions averaged 140 characters versus 97 in dev, and options 653 versus 117.
    • Accuracy fell by a similar amount for Beginner and Advanced items.
    • Questions involving sisters scored 0.644, while those involving spouses scored 0.794.
  • SubTask 2 breakdowns:
    • By level: Beginner 96.7%, Intermediate 95.3%, Advanced 78.7%.
    • By source book: from 63.6% (Fatḥ al-Mughīth) up to 100% on some books.
    • Errors had two causes: the relevant passage was missing from the retrieved set, or it was present but the model still chose wrongly.

Interesting ideas

  • I1. Instead of blending dense and lexical scores with a tuned weight, take a fixed top-k from each retriever, pool them, and let a cheap judge prune the pool; this avoids calibrating incompatible score scales.
  • I2. Force the model to wrap its answer in a fixed tag, extract it with a regex, and log every raw output beside the parsed label; this makes LLM output machine-checkable and auditable later.
  • I3. Diagnose a dev-to-test collapse by comparing input length distributions and by checking whether every difficulty level degrades equally; this separates a general distribution shift from a weakness in one category.
  • I4. Break down accuracy "inclusively", counting each item once under every entity type it mentions, which exposes which participants make problems hard.
  • I5. Split RAG errors into "evidence never retrieved" versus "evidence present but wrong choice", and report Wilson intervals against the chance baseline.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Candidate facts for a question could be gathered both by lemma/keyword match and by graph-neighbourhood lookup, then pooled and filtered by frame compatibility, with no fused score to tune.
  • I2 (learn): LLM teacher tiers could be required to emit silver labels (e.g. head/deprel or role) inside strict tags, keeping raw replies for repair-rule mining and disagreement analysis.
  • I3 (learn): When a parser or reader drops on a new set, compare sentence and option lengths and per-level deltas before blaming a specific construction.
  • I4 (answer): Maths word problem and fable QA errors could be tallied per entity/relation type in the world model (e.g. transfers versus comparisons) to see where reasoning breaks.
  • I5 (answer): Reader failures could be labelled as missing world facts versus wrong inference over present facts, with confidence intervals versus a random-option control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of KhartoumKhartoum · Sudan4 works · 71 citations🎓African Institute for Mathematical SciencesCape Town · South Africa8 works · 797 citations🎓African Institute for Mathematical SciencesKigali · Rwanda5 works · 444 citations🎓International Islamic University MalaysiaKuala Lumpur · Malaysia3 works · 4 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Yossra Noureldien, Hassan Suliman, Farah Attallah, Abdelrazig Mohamed and Sara Abdalla. 2025. Athar at QIAS2025: LLM-based Question Answering Systems for Islamic Inheritance and Classical Islamic Knowledge. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.126

@inproceedings{noureldien2025athar,
  title = {{Athar at QIAS2025: LLM-based Question Answering Systems for Islamic Inheritance and Classical Islamic Knowledge}},
  author = {Yossra Noureldien, Hassan Suliman, Farah Attallah, Abdelrazig Mohamed and Sara Abdalla},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.126},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.126}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The findings highlights that Subtask 1 requires stronger complex reasoning, whereas Subtask 2 would benefit from enhanced retrieval and LLM comprehension to achieve more reliable answer selection. — ArabicaQA: A Comprehensive Dataset for Arabic Question Answering
  2. Yousef Adel, Mostafa Dorrah, Ahmed Ashraf, Abdallah ElSaadany, Mahmoud Mohamed, Mariam Wael, and Ghada Khoriba. 2023. AraQA: An Arabic Generative Question-Answering Model for Authentic Religious Text. In 2023 11th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC), pages 235–239, Alexandria, Egypt. IEEE. find
  3. Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics.
  4. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydin. 2025. A RAG-based Question Answering System Proposal for Understanding Islam: MufassirQAS LLM. arXiv preprint. ArXiv:2401.15378 [cs]. ↗
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. ↗
  6. Khader I. Alkhouri. 2024. The Role of Artificial Intelligence in the Study of the Psychology of Religion. Religions, 15(3):290. ↗
  7. H. F. Alshahad and Z. A. Abutiheen. 2015. Computation of inheritance share in islamic law by an expert system using decision tables. Quarterly Adjudicated Journal for Natural and Engineering Research and Studies, 1:105–114. find
  8. Mitha Alshammary, Md Nahiyan Uddin, and Latifur Khan. 2024. RFPG: Question-Answering from Low-Resource Language (Arabic) Texts using Factually Aware RAG. In 2024 IEEE 10th International Conference on Collaboration and Internet Computing (CIC), pages 107–116, Washington, DC, USA. IEEE. find
  9. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390. ↗
  10. Mohamed Basem, Islam Oshallah, Baraa Hikal, Ali Hamdi, and Ammar Mohamed. 2025. Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models. In Faisal Saeed, Fathey Mohammed, Errais Mohammed, Shadi Basurra, and Mohammed Al-Sarem, editors, Advances on Intelligent Computing and Data Science II, volume 255, pages 244–254. Springer Nature Switzerland, Cham. Series Title: Lecture Notes on Data Engineering and Communications Technologies.
  11. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9. Association for Computational Linguistics. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Brij Gupta and Ivan Giannoccaro, editors. 2024. Challenges in Large Language Model Development and AI Ethics:. Advances in Computational Intelligence and Robotics. IGI Global.
  12. Zainal Habib. 2025. Ethics of Artificial Intelligence in Maqāṣid Al-Sharīa’s Perspective. KARSA Journal of Social and Islamic Culture, 33(1):105–134. ↗
  13. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547. find
  14. Zahra Khalila, Arbi Haza Nasution, Winda Monika, Aytug Onan, Yohei Murakami, Yasir Bin Ismail Radi, and Noor Mohammad Osmani. 2025. Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models. International Journal of Advanced Computer Science and Applications, 16(2). ↗
  15. Feng Liu, Jiaqi Jiang, Yating Lu, Zhanyi Huang, and Jiuming Jiang. 2025. The ethical security of large language models: A systematic review. Frontiers of Engineering Management, 12(1):128–140. ↗
  16. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  17. Qur’an QA 2022: Overview of The First Shared Task on Question Answering over the Holy Qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87, Marseille, France. European Language Resources Association.
  18. Faiza Qamar, Seemab Latif, and Asad Shah. 2024. Techniques, datasets, evaluation metrics and future directions of a question answering system. Knowledge and Information Systems, 66(4):2235–2268. ↗
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Preprint. find
  20. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. ↗
  21. Iqbal H. Sarker. 2024. LLM potentiality and awareness: a position paper from the perspective of trustworthy and responsible AI modeling. Discover Artificial Intelligence, 4(1):40. ↗
  22. Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, and Wadii Boulila. 2025. From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation. arXiv preprint. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/noureldien-2025-athar-qias2025-llm-based-question/