Elamur Research⌕
☰
home / papers / 2025 / Can LLMs Directly Retrieve Passages for Answering Questions from Qur'an?

Can LLMs Directly Retrieve Passages for Answering Questions from Qur'an?

Paper · 2025 · Sohaila Eltanbouly, Salam Albatarni, Shaimaa Hassanein et al.

Summary

This paper asks whether general-purpose chat LLMs can act as retrievers for the Qur'anic Passage Retrieval (QPR) task from the Qur'an QA 2023 shared task. In that task, a question in modern standard Arabic must be answered with a ranked list of at most ten Qur'anic passages, or with no answer at all. The authors tested seven models under eight prompt variants: zero-shot, chain-of-thought (reason via Tafseer), and few-shot with random or BM25-selected similar examples at 1–3 shots. They chose the best prompt for each model on a development split. Models never write scripture text. They return only a surah name and verse range, and post-processing maps these to official passages. On the test split, Claude-3.5-sonnet in zero-shot mode roughly doubled the previous best MAP and raised MRR by about 38 points. Every model except the small Arabic-centric Fanar beat the shared-task winners. The authors still regard absolute accuracy as too low for real religious use, and they do not endorse that use.

Key points

  • Data: the Qur'an QA 2023 QPR collection, with 1,266 topic-segmented passages, 251 questions and 1,599 question–passage pairs. The original 70/10/20 split was rebalanced to 40/40/20 because few-shot demonstrations needed only part of the training data.
  • Models: GPT-4o, Deepseek-V3, Claude-3.5-sonnet, Gemini-2.0-flash, Command R+, Mistral-large and Fanar (7B), all called through their official APIs at temperature 0. Evaluation used MAP and MRR at rank 10.
  • Best test results: Claude (ZS) reached MAP 0.535 and MRR 0.842, against 0.251 / 0.461 for the TCE cross-encoder ensemble. GPT-4o (FS-R, 3-shot) came second with 0.458 / 0.776. Fanar scored lowest at 0.156 / 0.295.
  • Sensitivity to prompts depended heavily on the model:
    • CoT helped only three models.
    • Command R+ moved just 2.1 MAP points across prompts, while Mistral swung by 13.6.
    • Claude and Fanar did not gain from few-shot prompting.
  • Failure analysis covered how many answers each model produced:
    • Claude averaged 8.2 answers per query, close to the 8.4 in the gold data.
    • GPT-4o never went above ten answers.
    • Gemini and Deepseek sometimes returned more than 50 answers.
    • For Claude, queries with more than ten relevant passages always scored AP below 0.6.

Interesting ideas

  • I1. Let the model output only references (chapter plus verse numbers), never the source text, and check them against the canonical source in a separate step; this blocks fabricated text and makes every answer checkable.
  • I2. Choose few-shot demonstrations by running BM25 over stored question+answer pairs, so each query receives the most lexically similar solved cases.
  • I3. Select the prompt separately for each model on a dev set rather than assuming one shared template, since the best configuration differed from model to model.
  • I4. Diagnose systems by comparing their answer-count distribution with the gold distribution, and by scoring how often an abstention is actually correct; this separates over-generation, over-caution and refusal.
  • I5. Plot per-query AP against the number of gold answers to see whether failures come from ranking or from limited recall.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): readers could cite sentence or entity node ids from the world graph instead of free text, then verify those ids exist and support the claim before replying.
  • I2 (learn): when asking an LLM teacher tier to adjudicate a disputed parse or frame, attach the nearest already-agreed silver examples found through cheap lexical retrieval.
  • I3 (learn): calibrate each teacher tier or witness with its own prompt and settings on a held-out set before trusting its silver labels.
  • I4 (answer): log answer counts and "no answer" precision on school tests and fable QA, treating questions with no answer as negative controls.
  • I5 (answer): break down QA scores by how many entities or events an answer needs, to find where world-of-text recall fails.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Qatar UniversityDoha · Qatar53 works · 1,027 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksQuestion answering · 487 worksInformation retrieval & search · 552 works

Cite

Cite as Sohaila Eltanbouly, Salam Albatarni, Shaimaa Hassanein and Tamer Elsayed. 2025. Can LLMs Directly Retrieve Passages for Answering Questions from Qur'an?. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.16

@inproceedings{eltanbouly2025can,
  title = {{Can LLMs Directly Retrieve Passages for Answering Questions from Qur'an?}},
  author = {Sohaila Eltanbouly, Salam Albatarni, Shaimaa Hassanein and Tamer Elsayed},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.16},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.16}
}

References

16 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydın. 2024. A RAG-based Question Answering System Proposal for Understanding Islam: MufassirQAS LLM. Arxiv preprint. ↗
  2. Hessa Alawwad, Lujain Alawwad, Jamilah Alharbi, and Abdullah Alharbi. 2023. AHJL at Qur’an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation. In Proceedings of ArabicNLP 2023, pages 702–707, Singapore (Hybrid). Association for Computational Linguistics.
  3. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Is GPT-4 a Good Islamic Expert for Answering Quran Questions? In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages XX–XX, Taipei City, Taiwan. The Association for Computational Linguistics and Chinese Language Processing. October 20-21. find
  4. Mohammed Alqarni. 2023. Embedding Search for Quranic Texts based on Large Language Models. International Journal on Islamic Applications in Computer Science and Technology, 4(4):20–29. ↗
  5. Mohamed Basem, Islam Oshallah, Baraa Hikal, Ali Hamdi, and Ammar Mohamed. 2024. Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models. Preprint, arXiv:2412.11431. ↗
  6. Mohammed Alaa Elkomy and Amany Sarhan. 2023. Tce at Qur’an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur’anic QA. In Proceedings of the First Arabic Natural Language Processing Conference (Arabic-NLP 2023), Singapore.
  7. B Hammo et al. 2002. QARAB: A: Question answering system to support the Arabic language. Proceedings of the ACL.
  8. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2024. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  9. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics. ↗
  10. Ghazaleh Mahmoudi, Yeganeh Morshedzadeh, and Sauleh Eetemadi. 2023. Gym at Qur’an QA 2023 Shared Task: Multi-Task Transfer Learning for Quranic Passage Retrieval and Question Answering with Large Language Models. In Quran QA 2023 Shared Task.
  11. Rana Malhas and Tamer Elsayed. 2020. AyaTEC: Building a Reusable Verse-based Test Collection for Arabic Question Answering on the Holy Qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing, 19(6):1–21. find
  12. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the Holy Qur’an using CL-AraBERT. Information Processing & Management, 59(6):103068. ↗
  13. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  14. Qur’an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur’an. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  15. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic Question Answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  16. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. Fanar: An Arabic-Centric Multimodal Generative AI Platform. arXiv preprint arXiv:2501.13944. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eltanbouly-2025-can-llms-directly-retrieve-passages/