Elamur Research⌕
☰
home / papers / 2025 / Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction

Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction

Paper · 2025 · Mohamed Basem; Islam Oshallah; Ali Hamdi; Khaled Shaban; and Hozaifa Kassab

Summary

The paper addresses question answering over the Qur'an. The task is hard because Classical Arabic differs from the Modern Standard Arabic that most models are trained on, the verses carry layered meaning, and annotated data is scarce. The authors work on both tracks of the Quran QA 2023 Shared Task: finding relevant passages and pulling exact answer spans from them.

They first enlarge the 251-question training set to more than 2,000 questions. They do this by paraphrasing and by merging outside resources: tafsir-based QA collections, a thematic plant list, verse–commentary pairs and general Arabic QA sets. For retrieval, they combine several fine-tuned Arabic language models into an ensemble. For span extraction, they skip fine-tuning on the small data and instead give instruction-tuned large language models a few worked examples in the prompt.

The system reports state-of-the-art scores on the shared task: MAP@10 0.3128 and MRR@10 0.5763 for retrieval, and pAP@10 0.669 for extraction. The work suggests that ensembling plus prompted LLMs is a practical recipe for narrow, low-resource QA domains.

Key points

  • Retrieval data is an extended AyaTECv1.2 over the Qur'anic Passage Collection (QPC). It has 1,266 topical passages and 251 questions, 37 of which have no answer, so the system must return nothing for them.
  • MRC data is an extended QRCDv1.2 with 1,562 question–passage pairs, 62 of them unanswerable. QUQA and ARCD were reformatted into the QRCDv1.2 schema so all sources could be trained on together.
  • Retrieval training was enriched with these sources:
    • "1000 QAs from the Holy Qur'an"
    • a list of plants in the Qur'an and Hadith
    • Tafsir al-Jalalayn verse–commentary pairs
    • the Arabic subset of TyDi QA
  • Reported results: MAP@10 0.3128, MRR@10 0.5763 (retrieval) and pAP@10 0.669 (extraction). The authors say these beat earlier shared-task systems; prior ensemble work they cite reached MAP@10 of about 0.34 with MRR of about 0.52.
  • The system retrieves whole thematic passages rather than single verses, to keep context intact. The authors also suggest that answers be framed as informational aids, not authoritative interpretation.

Interesting ideas

  • I1. Unanswerable questions are first-class cases: when the source holds no answer, the correct output is an empty result, and the benchmark scores this behaviour explicitly.
  • I2. Heterogeneous datasets are converted into one shared schema before training, so very different sources feed a single pipeline.
  • I3. With only a few hundred labelled examples, few-shot prompting of an instruction-tuned LLM beat fine-tuning for span extraction.
  • I4. The retrieval unit is a coherent thematic passage rather than an isolated sentence, because meaning depends on neighbouring text.
  • I5. Training questions are multiplied by systematic paraphrasing to cover more ways of asking the same thing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova's fable and school-test readers could be scored on correctly abstaining when the world of the text has no supporting fact, as a built-in negative control.
  • I2 (learn): Silver data from different witness parsers and teacher tiers could be normalised into one canonical record format before agreement voting and training.
  • I3 (learn): For rare frame types or new question styles, an LLM teacher prompted with a handful of examples could produce candidate labels, which Mova's explainable components then verify.
  • I4 (world): When locating support for an answer, Mova could pull the connected event span from the text's world (states, ordered events) rather than one isolated clause.
  • I5 (answer): Paraphrased variants of test questions could check that Mova's tree→frames→answer path gives the same result regardless of surface wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓October University of Modern Sciences and ArtsCairo · Egypt8 works · 25 citations🎓Qatar UniversityDoha · Qatar10 works · 81 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 3Research subjects and languages the work is about.Question answering · 242 worksSemitic languages · 1,484 worksInformation retrieval & search · 145 works

Cite

Cite as Mohamed Basem; Islam Oshallah; Ali Hamdi; Khaled Shaban; and Hozaifa Kassab. 2025. Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction. arXiv:2508.06971.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.06971 · arXiv 2508.06971

@misc{shaban2025two,
  title = {{Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction}},
  author = {Mohamed Basem; Islam Oshallah; Ali Hamdi; Khaled Shaban; and Hozaifa Kassab},
  year = {2025},
  doi = {10.48550/arxiv.2508.06971},
  eprint = {2508.06971},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.06971}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mattson, Ingrid. The story of the Qur’an: its history and place in Muslim life. John Wiley & Sons, 2012. find
  2. Utomo, FS, N. Suryana, and MS Azmi. “Question answering systems on holy quran: a review of existing frameworks, approaches, algorithms and research issues.” In Journal of Physics: Conference Series, vol. 1501, no. 1, pp. 012022, IOP Publishing, 2020. find
  3. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A Survey of Opinion Mining in Arabic: A Comprehensive System Perspective Covering Challenges and Advances in Tools, Resources, Models, Applications, and Visualizations. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 18, 3, Article 27 (September 2019), 52 pages. find
  4. Martinez-Gil, Jorge. “A survey on legal question–answering systems.” Computer Science Review, vol. 48, pp. 100552, 2023. ↗
  5. Wang, Dandan and Shiqing Zhang. “Large language models in medical and healthcare fields: applications, advances, and challenges.” Artificial Intelligence Review, vol. 57, no. 11, pp. 299, 2024. ↗
  6. Hamdi, Ali, Khaled Shaban, and Anazida Zainal. “A Review on Challenging Issues in Arabic Sentiment Analysis.” Journal of Computer Science, vol. 12, no. 9, Dec. 2016, pp. 471-481. ↗
  7. Altammami, Shatha and Eric Atwell. “Challenging the transformer-based models with a classical arabic dataset: Quran and hadith.In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 1462–1471,2022.
  8. Malhas, Rana and Tamer Elsayed. “AyaTEC: Building a Reusable Verse-Based Test Collection for Arabic Question Answering on the Holy Qur’an.” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 19, no. 6, pp. 1–21, November 2020. find
  9. Al-Khalifa, Hend S. and Nouf Alnazzawi. “QUQA: Question Answering Dataset for the Holy Qur’an.” 2023. find
  10. Seyyed Hossein Nasr, Caner K. Dagli, Maria Massi Dakake, Joseph E. B. Lumbard, and Mohammed Rustom, editors. The Study Quran: A New Translation and Commentary. HarperOne, New York, 2015. find
  11. Mohamed, M., Emad, R., & Hamdi, A. (2025). A Multi-Layered Large Language Model Framework for Disease Prediction. find
  12. A. Allam, S. Ahmed, A. Hamdi and A. Mohammed, ”Arabic Large Language Models for Medical Text Generation,” 2025 4th International Conference on Computer Technologies (ICCTech), Kuala Lumpur, Malaysia, 2025, pp. 1-6.
  13. Antoun, Wissam, Fady Baly, and Hazem Hajj. “AraBERT: Transformer-based Model for Arabic Language Understanding.” In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, pp. 9, 2020. ↗
  14. Antoun, Wissam, Fady Baly, and Hazem Hajj. “AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding.” In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 191–195, Association for Computational Linguistics, 2021. ↗
  15. Inoue, Go, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. “The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models.” In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Association for Computational Linguistics, April 2021.
  16. Khot, Tushar, Alon Talmor, Robin Jia, Peter Clark, Sewon Min, Eunsol Choi, and Hannaneh Hajishirzi. “MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension.” In Proceedings of the 2nd Workshop on Machine Reading for Question Answering (MRQA), pp. 1–13, 2019.
  17. ElKomy, Ahmed and Hadeer Sarhan. “TCE at Qur’an QA 2022: Arabic Language Question Answering Over Holy Qur’an Using a Post-Processed Ensemble of BERT-based Models.” find
  18. Ramachandran, Anand. “Unveiling Google’s Gemini 2.0: A Comprehensive Study of its Multimodal AI Design, Advanced Architecture, and Real-World Applications.” ResearchGate, 2024. find
  19. DeepSeek-AI, Aixin Liu, Bei Feng, and others. “DeepSeek-V3 Technical Report.” arXiv preprint arXiv:2412.19437, 2025. ↗
  20. Malhas, Mohammad, Hamdy Mubarak, and Kareem Darwish. “Qur’an QA 2023: Overview of the First Workshop on Question Answering on the Quran.” Proceedings of the 1st Workshop on Question Answering on the Quran, pp. 1–13, October 2023. find
  21. Elkomy, Abdelrahman, Mahmoud Elsayed, and Mohamed Hefny. “Tce team: Transformer-based ensemble for extractive qur’anic qa.” In Proceedings of the 1st Workshop on Question Answering on the Quran, pp. 110–117, 2023. find
  22. Alawwad, Hessa, Lujain Alawwad, Jamilah Alharbi, and Abdullah Alharbi. “Ahjl at qur’an qa 2023 shared task: Enhancing passage retrieval using sentence transformer and translation.” In Proceedings of ArabicNLP 2023, pp. 702–707, 2023. find
  23. Alnefaie, Raghad, Nouf Alghamdi, Atheer Bajawi, Abeer Aljohani, Areej Barnawi, Fatimah Daghistani, and Abdullah Awajan. “Lkau23 team: Transformer-based ensemble for qur’anic qa.” In Proceedings of the 1st Workshop on Question Answering on the Quran, pp. 141–147, 2023. find
  24. Zekiye, Muhammed, Mohamed Bakouri, Mariam Oudah, and Alaa Mourad. “Al-jawaab team: Generative constraint modeling with dense passage retrieval for quranic question answering.” In Proceedings of the 1st Workshop on Question Answering on the Quran, pp. 126–133, 2023. find
  25. Alkhurayyif, Yazeed and Abdul Rahaman Wahab Sait. “Developing an Open Domain Arabic Question Answering System Using a Deep Learning Technique.” IEEE Access, vol. 11, pp. 79659–79671, 2023. ↗
  26. Durachman, A. and A. Adriani. “IndoQA: A Question Answering System for Indonesian Factoid Questions.” In Proceedings of the 2019 International Conference on Asian Language Processing (IALP), pp. 329–334, IEEE, 2019. find
  27. Bhaskar, Prashant, Hammad Sheikh, and Nasimul Majumder. “SIEG at MEDIQA 2019: Multi-task Neural Ensemble for Biomedical Question Answering.” In Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 398–403, 2019. find
  28. Abdullah Saeed. Interpreting the Qur’an: Towards a Contemporary Approach. Routledge, London and New York, 2005. find
  29. Ashor, Q. “1000 QAs from the Holy Qur’an.” Noor Book, 2023. find
  30. Qur’anic Botanic Garden (QBG). “List of Plants Citation in Qur’an and Hadith v5.” Qur’anic Botanic Garden Publication, 2021. find
  31. Al-Mahalli, Jalal al-Din and Jalal al-Din al-Suyuti. “Tafsir al-Jalalayn.” Interactive Tafsir Platform, 2021. find
  32. Clark, Jonathan H., Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. “Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages.” Transactions of the Association for Computational Linguistics, vol. 8, pp. 454–470, 2020. ↗
  33. M. Basem, I. Oshallah, B. Hikal, A. Hamdi, and A. Mohamed, “Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models,” in Proc. Int. Conf. Advanced Computing and Informatics, pp. 244–254, Springer, 2024.
  34. Oshallah, I., Basem, M., Hamdi, A. and Mohammed, A., 2025. Cross-Language Approach for Quranic QA. arXiv preprint arXiv:2501.17449. ↗
  35. Malhas, Rana and Tamer Elsayed. “Arabic Machine Reading Comprehension on the Holy Qur’an using CL-AraBERT.” Information Processing & Management, vol. 59, no. 6, pp. 103068, 2022. find
  36. TheDevastator. “ARCD Dataset: Unlocking Arabic Language Comprehension with the Arabic Reading Comprehension Dataset.” 2021. find
  37. Basem, Mohamed, Islam Oshallah, Ali Hamdi, and Ammar Mohammed. “Few-Shot Prompting for Extractive Quranic QA with Instruction-Tuned LLMs.” arXiv preprint arXiv:2508.06103, 2025. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/basem-2025-two-stage-quranic-qa-ensemble/