Elamur Research⌕
☰
home / papers / 2025 / SEA-Team at QIAS 2025: Enhancing LLMs for Question Answering in Islamic Texts

SEA-Team at QIAS 2025: Enhancing LLMs for Question Answering in Islamic Texts

Paper · 2025 · Sanaa Alowaidi

Summary

This entry to the QIAS 2025 shared task covers two subtasks. Task 1 is Islamic inheritance reasoning: six-option multiple-choice questions on dividing an estate under Islamic jurisprudence. Task 2 tests general Islamic knowledge with four-option questions spanning the Prophet's life story, the principles of jurisprudence, the study of the Quran, and the assessment of Hadith reports. The authors build a retrieval-augmented generation (RAG) pipeline in LangChain:

  • documents are cut into 500-token chunks with a 50-token overlap;
  • chunks are embedded with text-embedding-3-large and indexed in FAISS;
  • the top four chunks go to the LLM together with the question and its options.

The knowledge sources are a fatwa collection for Task 1, which they also enlarge with training question–answer pairs, and roughly 50 Islamic books for Task 2. They compare plain similarity search with two keyword-driven retrieval variants, K2R and MQR-K, across five LLMs, some general multilingual and some Arabic-centric. The general models win. Mistral-large reaches 72.0% on Task 1 development data, and GPT-4o reaches 87.71% on Task 2. The study shows that simple retrieval and knowledge-base choices change accuracy as much as swapping the model does.

Key points

  • Data: 22,000 inheritance MCQs (20k train / 1k dev / 1k test, beginner and advanced) plus 3,165 fatwas. Task 2 has 1,700 MCQs (700 dev / 1,000 test, three difficulty levels) plus about 50 Islamic books.
  • Models compared: GPT-4o, Mistral-large-latest, Fanar Islamic-RAG, ALLaM-7B and Mistral-SABA-24B. Three-shot prompting serves as the baseline.
  • Retrieval variants:
    • K2R retrieves with the original query and with GPT-4o-extracted keywords in parallel, then merges and deduplicates the results.
    • MQR-K turns each keyword plus the query into an Arabic sub-question and retrieves for each one.
    • Keywords pass a filter that removes stop words, short terms and duplicates, and drops anything with cosine similarity below 0.3 to the question.
  • Task 1 dev results: adding training QA pairs to the fatwa base raised every model. Mistral-large went from 66.0 to 72.0 and Fanar from 57.8 to 62.8. On the test set, Mistral-large with K2R scored 63.0.
  • Task 2 results: GPT-4o with K2R scored 87.71 on dev. On test, K2R gave about 89% for both Mistral-large (89.1) and GPT-4o (89.0). MQR-K mostly hurt performance; Fanar was the exception.

Interesting ideas

  • I1. Labelled training items (question plus correct answer) are indexed as retrievable context, which turns supervision data into a knowledge base. This was the single most reliable gain on the reasoning task.
  • I2. Query expansions are pruned by checking their semantic similarity to the original question before use, which keeps added terms from drifting off-topic.
  • I3. Broader retrieval helped weaker models but distracted stronger ones. This was especially true in inheritance texts, where the same kinship words recur everywhere, so recall and precision should be tuned per model and per domain.
  • I4. A post-generation validator forces the output into a single option letter before scoring, which separates format errors from reasoning errors.
  • I5. Configurations chosen on dev did not keep their ranking on test: Mistral-large with K2R gained on test while GPT-4o with K2R slipped. Several candidate setups are therefore worth carrying forward to held-out evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): solved school-test items and word problems could be stored as citable precedents, so a reader explains its answer by pointing to a structurally similar solved case.
  • I2 (world): when a question is expanded through WordNet/FrameNet/Wikidata neighbours, expansions far from the question's frame could be discarded so the knowledge lookup stays focused.
  • I3 (answer): fable and tale QA with recurring characters could prefer narrow, event-anchored evidence over wide keyword matches, and a negative control could measure when extra context starts hurting.
  • I4 (learn): outputs from the LLM teacher tiers could pass a strict schema check before being admitted as silver labels, so malformed answers never contaminate training.
  • I5 (learn): two or three competing parser or repair-rule configurations could be kept through to the held-out test instead of committing to the single best dev score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LeedsLeeds · United Kingdom40 works · 1,822 citations🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 works

Cite

Cite as Sanaa Alowaidi. 2025. SEA-Team at QIAS 2025: Enhancing LLMs for Question Answering in Islamic Texts. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.130

@inproceedings{alowaidi2025sea,
  title = {{SEA-Team at QIAS 2025: Enhancing LLMs for Question Answering in Islamic Texts}},
  author = {Sanaa Alowaidi},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.130},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.130}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur A. Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed K. Elmagarmid, Mohamed Y. Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, and 22 others. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. ↗
  2. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydın. 2025. Improving llm reliability with rag in religious question-answering: Mufassirqas. Turkish Journal of Engineering, 9(3):544–559. ↗
  3. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language models. Preprint, arXiv:2311.16867. ↗
  4. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Is GPT-4 a good islamic expert for answering Quran questions? In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages 124– 133, Taipei City, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). find
  5. Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
  6. Binglan Han, Teo Susnjak, and Anuradha Mathrani. 2024. Automating systematic literature reviews with retrieval-augmented generation: a comprehensive overview. Applied Sciences, 14(19):9103. ↗
  7. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.
  8. Zhicong Li, Jiahao Wang, Zhishu Jiang, Hangyu Mao, Zhongxia Chen, Jiazhen Du, Yuanxing Zhang, Fuzheng Zhang, Di Zhang, and Yong Liu. 2024. Dmqr-rag: Diverse multi-query rewriting for rag. arXiv preprint arXiv:2411.13154. ↗
  9. Agada Joseph Oche, Ademola Glory Folashade, Tirthankar Ghosal, and Arpan Biswas. 2025. A systematic review of key retrieval-augmented generation (rag) systems: Progress, gaps, and future directions. arXiv preprint arXiv:2507.18910. ↗
  10. Sarah Alnefaie, Eric Atwell, and Mohammed Ammar Alsalka. 2024. Using the retrieval-augmented generation technique to improve the performance of gpt-4 in answering quran questions. In 2024 6th International Conference on Natural Language Processing (ICNLP), pages 377–381. find
  11. Mohammad Amaan Sayeed, Mohammed Talha Alam, Raza Imam, Shahab Saquib Sohail, and Amir Hussain. 2025. From rag to agentic: Validating islamic-medicine responses with llm agents. arXiv preprint arXiv:2506.15911. ↗
  12. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390. ↗
  13. Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2024. Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17716–17736, Miami, Florida, USA. Association for Computational Linguistics. ↗
  14. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics.
  15. Qimin Yang, Huan Zuo, Runqi Su, Hanyinghong Su, Tangyi Zeng, Huimei Zhou, Rongsheng Wang, Jiexin Chen, Yijun Lin, Zhiyi Chen, and Tao Tan. 2025. Dual retrieving and ranking medical large language model with retrieval augmented generation. Scientific Reports, 15(1):18062. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed A comprehensive description of the proposed RAG system is illustrated in Figure 1. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alowaidi-2025-sea-team-qias-2025-enhancing/