Elamur Research⌕
☰
home / papers / 2025 / SHA at the QIAS Shared Task: LLMs for Arabic Islamic Inheritance Reasoning

SHA at the QIAS Shared Task: LLMs for Arabic Islamic Inheritance Reasoning

Paper · 2025 · Shatha Altammami

Summary

This system paper addresses SubTask 1 of the QIAS 2025 Shared Task: answering Arabic multiple-choice questions on ilm al-mawārīth (Islamic inheritance law). Each question describes a set of heirs, and the model must pick the correct share distribution, which requires exact multi-step legal reasoning.

The author first set two baselines on the development set: a fine-tuned AraBERTv2 and Fanar prompted with two examples. She then compared three prompting setups on Gemini, GPT-4o, LLaMA-3.3-70b, Fanar and a majority-vote ensemble:

  • static five-shot examples,
  • dynamic five-shot examples chosen by similarity to the question,
  • dynamic examples plus retrieval of the closest fatwa answer from an IslamWeb corpus.

Retrieval uses a weighted, progressive n-gram overlap instead of embeddings. Embedding search mixed up legally distinct heirs, such as treating a son and a daughter as similar.

Gemini with dynamic examples and RAG scored best on the test set at 62.26%. GPT-4o, by contrast, got worse as more context was added. The work shows that whether retrieved context helps depends on the model, and that lexical matching can beat semantic vectors in precise legal domains.

Key points

  • Data:
    • 20,000 training MCQs, a 1,000-item validation set (beginner, intermediate and advanced) and a 1,000-item test set (500 beginner, 500 advanced, no intermediate).
    • Each question has 4–6 options with exactly one correct answer.
    • A supplementary corpus of 3,165 IslamWeb fatwas, each a question plus an expert answer.
  • Baselines (dev accuracy): fine-tuned AraBERTv2 reached 47.4%; Fanar with two-shot prompting reached 49.7%.
  • Retrieval scoring:
    • Trigram, bigram and unigram overlaps carry weights of 1.0, 0.5 and 0.2.
    • N-grams matched at a longer order are excluded from later counting.
    • The top 5 training MCQs are used as examples, and the top 1 fatwa answer is used as context.
  • Decoding: temperature was set to 0 and output was capped at 2 tokens, so the model returns just the option letter.
  • Results:
    • Gemini, Trial 3: 61.80 dev / 62.26 test.
    • Ensemble, Trial 3: best dev score at 63.40 but only 57.30 on test.
    • GPT-4o test accuracy fell from 55.55 to 47.90 to 46.30 across the three trials.

Interesting ideas

  • I1. Embedding similarity blurred heirs whose legal roles differ, so the author used weighted lexical n-gram overlap, where a single differing role word changes the match; this suits any domain where small contrasting words decide the answer.
  • I2. The progressive scheme counts long n-gram matches first and removes them before scoring shorter ones, which rewards specific phrase matches without double-counting their words.
  • I3. One retrieval function serves two jobs, choosing per-question few-shot examples and fetching a supporting expert document, which keeps the pipeline small and consistent.
  • I4. Context that helped one model hurt another (Gemini improved, GPT-4o fell), so added retrieval must be checked per model rather than assumed beneficial.
  • I5. Ensemble gains seen on dev shrank on test because the two sets had different difficulty mixes, a reminder to compare split composition before trusting dev improvements.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Retrieving analogous cases or knowledge-graph facts for kinship, quantity or role-sensitive questions should key on exact role lemmas rather than vector closeness, so "son" and "daughter" never count as interchangeable.
  • I2 (answer): A cheap, explainable retriever for choosing similar solved word problems or fable QA examples could use this cascade, and each match could be justified by listing the shared phrases.
  • I3 (learn): When querying LLM teacher tiers, the same similarity routine could pick the most similar silver-annotated sentences as demonstrations and also attach relevant reference material.
  • I4 (learn): Teacher-tier evaluation should test each teacher with and without extra context as a control, so that distraction-prone teachers do not feed noise into silver data.
  • I5 (learn): Evaluation reports for parsers or readers should break scores down by difficulty or genre bucket, so a mismatch between dev and test slices is visible early.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Saud UniversityRiyadh · Saudi Arabia37 works · 527 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 works

Cite

Cite as Shatha Altammami. 2025. SHA at the QIAS Shared Task: LLMs for Arabic Islamic Inheritance Reasoning. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.119

@inproceedings{altammami2025sha,
  title = {{SHA at the QIAS Shared Task: LLMs for Arabic Islamic Inheritance Reasoning}},
  author = {Shatha Altammami},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.119},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.119}
}

References

19 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shatha Altammami, Eric Atwell, and Ammar Alsalka. 2019. Text segmentation using n-grams to annotate hadith corpus. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 31–39. find
  2. Wissam Antoun and 1 others. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC.
  3. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Second Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics.
  4. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Md.
  5. Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and M. Abdul-Mageed. 2023. Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. ArXiv, abs/2305.14976. ↗
  6. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199– 22213. Curran Associates, Inc.
  7. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474.
  8. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics. ↗
  9. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  10. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79– 87, Marseille, France. European Language Resources Association.
  11. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2023. A ArXiv, survey of large language models. abs/2303.18223. ↗
  12. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  13. Qur’an QA 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an. In Proceedings of ArabicNLP 2023, pages 690–701, Singapore (Hybrid). Association for Computational Linguistics.
  14. Andrea Matarazzo and Riccardo Torlone. 2025. A survey on large language models with some insights on their capabilities and limitations. ArXiv, abs/2501.04040. ↗
  15. A. Plaat, Annie Wong, Suzan Verberne, Joost Broekens, N. V. Stein, and T. Back. 2024. Reasoning with large language models, a survey. ArXiv, abs/2407.11511. ↗
  16. Yash Saxena, Sarthak Chopra, and Arunendra Mani Tripathi. 2024. Evaluating consistency and reasoning capabilities of large language models. In 2024 Second International Conference on Data Science and Information System (ICDSIS), pages 1–5. IEEE. ↗
  17. Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics, 11:1–17. ↗
  18. Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, and 1 others. 2024. Retrieval-augmented generation for natural language processing: A survey. arXiv preprint arXiv:2407.13193. ↗
  19. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024, pages 6233–6251.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/altammami-2025-sha-qias-shared-task-llms/