Elamur Research⌕
☰
home / papers / 2025 / NUR at IslamicEval 2025 Shared Task: Retrieval-Augmented LLMs for Qur'an and Hadith QA

NUR at IslamicEval 2025 Shared Task: Retrieval-Augmented LLMs for Qur'an and Hadith QA

Paper · 2025 · Serag Amin, Ranwa Aly, Yara Allam et al.

Summary

The paper describes the NUR team's entry for subtask 2 of IslamicEval 2025. The task is to take a Modern Standard Arabic question and return a ranked list of up to 20 Qur'anic passages or Hadiths that may answer it. Systems must also detect when no answer exists. The search space has two parts: the Thematic Qur'anic Passage collection (QPC) and Sahih Al-Bukhari. Training data is scarce, since AyaTEC v1.3 covers only the Qur'an and no Hadith QA set exists.

The system has two stages. First, a dense retriever based on AraModernBert is fine-tuned with hard negatives, either directly or through a multi-stage curriculum of external Arabic QA data. Second, the retrieved candidates are reranked by either a cross-encoder (GATE, AraBERT) or Gemini 2.5 Flash. Questions are flagged as unanswerable when every reranked score falls under a threshold.

The curriculum clearly improved recall. Gemini reranking scored best on test, with MAP@10 0.1809, but test scores dropped well below development results. The study shows what helps and what hurts in a low-resource, multi-corpus religious QA setting.

Key points

  • Corpora:
    • QPC holds 1,266 thematic passages without diacritics.
    • The Sahih Al-Bukhari collection holds 2,254 Hadiths.
    • AyaTEC v1.3 has 210 train, 40 dev and 71 test questions; about 15% of train and dev questions have zero answers.
  • Retriever baseline and pipelines:
    • AraModernBert-Base-STS was the best off-the-shelf retriever, with Recall@30 0.445.
    • Pipeline A fine-tunes once on the task data and reaches R@30 0.491 and R@70 0.592.
    • Pipeline B is a curriculum (TyDiQA → Jalalayn Tafseer → QuQA → HaQA → task set) and reaches R@30 0.541 and R@70 0.688.
  • Cross-encoder fine-tuning hurt: the untuned GATE reranker had the best dev MAP@5 (0.3172), and every fine-tuning step lowered it. AraBERT, by contrast, gained a lot from fine-tuning.
  • Two enhancements made things worse:
    • LDA/LLM-based topic pre-filtering over 40 Islamic topics cost more than 2% MAP.
    • Adding Tafseer text to passages before LLM reranking capped MAP@10 at roughly 0.15.
  • Test results:
    • Gemini on a combined top-70 pool scored MAP@10 0.1809, MAP_Q@5 0.2334 and MAP_H@5 0.1923.
    • A variant with a 50 Qur'an / 20 Hadith split scored about the same.
    • GATE (Pipeline B1) scored lower, with MAP@10 0.1257.

Interesting ideas

  • I1. Curriculum ordering: Training moved step by step from generic Arabic QA to religious-domain QA, and this beat training on the small target set alone, by up to about 10% recall at deep cut-offs.
  • I2. Bounded hard-negative window: Hard negatives were drawn from the base model's own top-K results, and a window of about 60–80 worked best; larger windows let in easy negatives that diluted the training signal.
  • I3. Threshold-based abstention: A question was declared unanswerable when no reranked candidate scored above a set threshold, which gives an explicit "no answer" output instead of a forced guess.
  • I4. Per-source quotas: Candidates were drawn with a fixed split between the Qur'an and Hadith corpora so that the smaller, harder source was never crowded out of the pool.
  • I5. Check the untuned baseline first, and watch input length: Further training of an already strong reranker reduced quality, and adding more context to the LLM (more passages or commentary) also lowered scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's parsers and readers could be trained in stages, first on broad silver data and then on narrower domain sets such as fables or school texts, rather than mixing everything at once.
  • I2 (learn): When building contrastive or ranking examples for gparse or frame selection, take negatives from the model's own near-misses and cap how many are used.
  • I3 (answer): Fable and test readers could return a principled "not stated in the text" when no world-state match passes a confidence bar, and that bar could be tuned on known zero-answer items.
  • I4 (world): When looking up compiled knowledge, reserving slots for each source (WordNet, FrameNet, Wikidata) would keep the sparser source from being drowned out.
  • I5 (learn): Every fine-tuning or LLM-teacher step should be compared against the frozen model as a negative control, and longer teacher prompts should be tested rather than assumed to help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations🎓Nile UniversityGiza · Egypt14 works · 158 citations🎓Nile UniversityArua · Uganda4 works · 4 citations🎓Nile UniversityKhartoum · Sudan2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Serag Amin, Ranwa Aly, Yara Allam, Yomna Eid and Ensaf Hussein Mohamed. 2025. NUR at IslamicEval 2025 Shared Task: Retrieval-Augmented LLMs for Qur'an and Hadith QA. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.68

@inproceedings{amin2025nur,
  title = {{NUR at IslamicEval 2025 Shared Task: Retrieval-Augmented LLMs for Qur'an and Hadith QA}},
  author = {Serag Amin, Ranwa Aly, Yara Allam, Yomna Eid and Ensaf Hussein Mohamed},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.68},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.68}
}

References

22 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. B. A Al-Sharjy and Z. Al-Zubaidi. 2009. Al-Tajreed Al-Sareeh of Collective Sahih Hadith. . find
  2. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Haqa and quqa: Constructing two arabic question-answering corpora for the quran and hadith. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 90–97.
  3. Nikolaos Ampazis. 2024. Improving RAG quality for large language models with topic-enhanced reranking. In Proceedings of the 2024 IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI 2024), pages 74–87, Cham. Springer Nature Switzerland. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  5. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. ↗
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, and et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
  7. Mohammed ElKomy and Amany Sarhan. 2023. Tce at qur’an qa 2023 shared task: Low resource enhanced transformer-based ensemble approach for qur’anic qa. In Proceedings of the Qur’an QA 2023 Shared Task. Tanta University, Egypt.
  8. Marwan N. Swar. 2007. Mushaf Al-Tafseel Al Mawdoo’ee. Dar Al-Fajr Al-Islami, Damascus. find
  9. Fajri Koto, Haonan Li, Sara Shatanawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. Arabicmmlu: Assessing massive multitask language understanding in arabic. In Findings of the Association for Computational Linguistics: ACL 2024.
  10. Xianming Li and Jing Li. 2023. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871. ↗
  11. Rana Malhas and Tamer Elsayed. 2020. Ayatec: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6). find
  12. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  13. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87, Marseille, France. European Language Resources Association.
  14. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  15. Qur’an QA 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an. In Proceedings of ArabicNLP 2023, pages 690–701, Singapore (Hybrid). Association for Computational Linguistics.
  16. Hamza Shahid Mohaned A. Rashad. 2025. The arabic rag leaderboard. urlhttps://huggingface.co/spaces/Navid-AI/The-Arabic-Rag-Leaderboard. ↗
  17. Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Darwish, and Walid Magdy. 2025. IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  18. Omer Nacar and Anis Koubaa. 2024. Enhancing semantic similarity understanding in arabic nlp with nested embedding learning. ↗
  19. Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, and Wadii Boulila. 2025. Gate: General arabic text embedding for enhanced semantic textual similarity with matryoshka representation learning and hybrid loss training. ↗
  20. NAMAA-Space. 2025. Gate-reranker-v1. https://huggingface.co/NAMAA-Space/ GATE-Reranker-V1. Hugging Face model, Apache-2.0 license. Accessed: 15 August 2025. ↗
  21. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  22. Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, pages 1503–1512, New York, NY, USA. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/amin-2025-nur-islamiceval-2025-shared-task/