Elamur Research⌕
☰
home / papers / 2026 / From RAG to Agentic RAG for Faithful Islamic Question Answering

From RAG to Agentic RAG for Faithful Islamic Question Answering

Paper · 2026 · Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar et al.

Summary

LLMs are increasingly used to answer Islamic questions, where an unsupported answer can have real religious consequences. Multiple-choice and reading-comprehension benchmarks cannot show two key failures: free-form hallucination and the model's ability to decline when evidence is lacking. The authors build ISLAMICFAITHQA, a generative Arabic/English benchmark of 3,810 items. Each item has one atomic gold answer, and outputs are graded by an LLM judge as correct, incorrect or not attempted.

They also release a modeling suite:

  • supervised reasoning pairs grounded in Quran and Hadith text,
  • bilingual preference data for reward-guided alignment,
  • a Quran retrieval corpus split into individual verses.

Using these resources, they evaluate several configurations: the plain base model, versions with SFT, with RL and with retrieval augmentation, and an agentic retrieval setup. In the agentic setup, the model calls tools to search, read verses and look up metadata before it answers.

Most off-the-shelf models stay under 30% accuracy. Retrieval helps consistently, and the agentic loop gives the biggest gains. With it, a 4B model rivals much larger systems, and the gap between its Arabic and English scores largely closes. This points toward more trustworthy assistants for this high-stakes domain.

Key points

  • ISLAMICFAITHQA has 3,810 bilingual questions. Grading is strict and three-way (correct / incorrect / not attempted), so hallucination and abstention can be measured directly.
  • The resources are:
    • 25K text-grounded SFT reasoning pairs, LLM-generated from Quran and Hadith;
    • 5K bilingual preference samples intended for GRPO-style training;
    • 6,236 ayat with surah/ayah indices, canonical IDs and normalized Arabic.
  • Qwen3-4B-2507 accuracy rises from 21.85 (base) to 38.85 (+RAG) to 48.90 (+Agentic RAG). Its Arabic/English split moves from 15.80/27.90 to 49.60/48.20. The best overall result is Fanar-2-27B + Agentic RAG at 57.30.
  • Error analysis covers 36 questions that standard RAG failed and agentic RAG solved. In 24 of them the model had previously abstained; in 12 it had answered wrongly.
  • The cost for Qwen3-4B: about 6 tool calls per query, tokens up from 502 to 1,520, and latency up from 0.52s to 3.45s (6.63×).

Interesting ideas

  • I1. Grading with a separate "not attempted" label keeps a cautious refusal distinct from a confident wrong answer, so a system's honesty and its accuracy become visible separately.
  • I2. Restricting benchmark items to atomic questions with a single gold answer makes open-ended generation gradable reliably, at the price of excluding contested cases.
  • I3. The knowledge base is cut into the smallest citable units, each with a canonical ID, index metadata and normalized text, so retrieval and citation checking line up exactly.
  • I4. Instead of retrieving once, the model loops through search, reading a specific unit, refining its query and revising its answer. This fixes both unanswered and wrongly answered questions.
  • I5. Comparing two systems by counting which items moved from which label to correct shows the mechanism behind a gain, not just its size.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test and fable-QA readers could be allowed to abstain when the world model lacks support, with abstentions scored apart from errors so that guessing is not rewarded.
  • I2 (learn): When building evaluation sets for free-text answers, pick items with one checkable answer so automatic grading stays trustworthy, and record disputed items separately.
  • I3 (world): Compiled knowledge entries (WordNet/Wikidata facts, verb-class records) could carry stable IDs, so every answer cites the exact fact it relied on.
  • I4 (answer): A reader could query the text world repeatedly (find a candidate entity, inspect its states and events, re-query) before committing, with each step logged for explainability.
  • I5 (learn): Evaluation runs could report transition tables between parser or reader versions, separating new answers from repaired errors and catching regressions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🔬Arab Center for Research and Policy StudiesDoha · Qatar8 works · 69 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz al-Khatib, Logan Cochrane, Kareem Mohamed Darwish, Rashid Yahiaoui and Firoj Alam. 2026. From RAG to Agentic RAG for Faithful Islamic Question Answering. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1317

@inproceedings{bhatia2026rag,
  title = {{From RAG to Agentic RAG for Faithful Islamic Question Answering}},
  author = {Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz al-Khatib, Logan Cochrane, Kareem Mohamed Darwish, Rashid Yahiaoui and Firoj Alam},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1317},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1317}
}

References

49 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  2. Alok Abhishek, Lisa Erickson, and Tushar Bandopadhyay. 2025. Beats: Bias evaluation and assessment test suite for large language models. 2503.24310v1. ↗
  3. Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, and Monojit Choudhury. 2024. Ethical reasoning and moral value alignment of llms depend on the language we prompt them in. 2404.18460v1. find
  4. Aisha Alansari and Hamzah Luqman. 2025. AraHalluE-val: A fine-grained hallucination evaluation framework for Arabic LLMs. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 148–161, Suzhou, China. Association for Computational Linguistics. find
  5. Hayfa A Aleid and Aqil M Azmi. 2025. Hajj-fqa: A benchmark arabic dataset for developing question-answering systems on hajj fatwas: H. aleid and a. azmi. Journal of King Saud University Computer and Information Sciences, 37(6):135. ↗
  6. Hamza Aljaji, Rawan Mohamed, Roaa Ibrahim, Abdallah Alkanani, Arwa Abdulhakim Elaradi, and Ehsaneddin Asgari. 2025. Benchmarking generative ai on quranic knowledge. In Proceedings of the 5th Muslims in ML Workshop at NeurIPS 2025. find
  7. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025. PalmX 2025: The first shared task on benchmarking LLMs on Arabic and islamic culture. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 774–789, Suzhou, China. Association for Computational Linguistics.
  8. Bushra Asseri, Estabrag Abdelaziz, and Areej Al-Wabil. 2025. Prompt engineering techniques for mitigating cultural bias against arabs and muslims in large language models: A systematic review. 2506.18199v2. 26478 10 find
  9. Farah Atif, Nursultan Askarbekuly, Kareem Darwish, and Monojit Choudhury. 2025. Sacred or synthetic? evaluating llm reliability and abstention for religious questions. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 217–226. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, and 7 others. 2025. AL-Lam: Large language models for arabic and english. In The Thirteenth International Conference on Learning Representations.
  10. Muhammad Huzaifa Bashir, Aqil M Azmi, Haq Nawaz, Wajdi Zaghouani, and Mona Diab. 2021. Arabic natural language processing for qur’anic research: a systematic review. Artificial Intelligence Review, 56(Suppl 1):13951–13993. ↗
  11. Gagan Bhatia, El Moatez Billah Nagoudi, Abdellah El Mekki, Fakhraddin Alwajih, and Muhammad Abdul-Mageed. 2025. Swan and ArabicMTEB: Dialect-aware, Arabic-centric, cross-lingual, and cross-cultural embedding models and benchmarks. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4654–4670, Albuquerque, New Mexico. Association for Computational Linguistics.
  12. Abdessalam Bouchekif, Samer Rashwani, Emad Soliman Ali Mohamed, Mutaz Alkhatib, Heba Sbahi, Shahd Gaben, Wajdi Zaghouani, Aiman Erbad, and Mohammed Ghaly. 2025a. QIAS 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 851–860, Suzhou, China. Association for Computational Linguistics.
  13. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. 2509.01081v2.
  14. Passant Elchafei and Mervet Abu-Elkheir. 2025. Span-level hallucination detection for llm-generated answers. 2504.18639v1. ↗
  15. Fatma Elsafoury and David Hartmann. 2025. Out of sight out of mind, out of sight out of mind: Measuring bias in language models against overlooked marginalized groups in regional contexts. 2504.12767v1. ↗
  16. Xiyan Fu and Wei Liu. 2025. How reliable is multilingual LLM-as-a-judge? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11040–11053, Suzhou, China. Association for Computational Linguistics. find
  17. Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, and Ethan Perez. 2025. Inverse scaling in test-time compute. 2507.14417v1. ↗
  18. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  19. Geyang Guo, Tarek Naous, Hiromi Wakaki, Yukiko Nishimura, Yuki Mitsufuji, Alan Ritter, and Wei Xu. 2025. CARE: Multilingual human preference learning for cultural awareness. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32854–32883, Suzhou, China. Association for Computational Linguistics. find
  20. Lukas Haas, Gal Yona, Giovanni D’Antonio, Sasha Goldshtein, and Dipanjan Das. 2025. Simpleqa verified: A reliable factuality benchmark to measure parametric knowledge. Preprint, arXiv:2509.07968. ↗
  21. Hunzalah Hassan Bhatti, Youssef Ahmed, Md Arid Hasan, and Firoj Alam. 2025. Cul-tranAI at PalmX 2025: Data augmentation for cultural knowledge representation. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 809–817, Suzhou, China. Association for Computational Linguistics. find
  22. Chang Hong, Minghao Wu, Qingying Xiao, Yuchi Wang, Xiang Wan, Guangjun Yu, Benyou Wang, and Yan Hu. 2025. Towards assessing medical ethics from knowledge to practice. 2508.05132v1. find
  23. Mohammad Hosseini, Kimia Hosseini, Shayan Bali, Zahra Zanjani, and Saeedeh Momtazi. 2025. Perhal-lueval: Persian hallucination evaluation benchmark for large language models. 2509.21104v1. find
  24. Yue Huang, Qihui Zhang, Philip S. Y, and Lichao Sun. 2023. Trustgpt: A benchmark for trustworthy and responsible large language models. 2306.11507v1. ↗
  25. Zheng Hui, Yijiang River Dong, Ehsan Shareghi, and Nigel Collier. 2025. TRIDENT: Benchmarking llm safety in finance, medicine, and law. arXiv preprint arXiv:2507.21134. ↗
  26. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  27. Junfeng Jiao, Saleh Afroogh, Abhejay Murali, Kevin Chen, David Atkinson, and Amit Dhurandhar. 2025. Llm ethics benchmark: A three-dimensional assessment system for evaluating moral reasoning in large language models. 2505.00853v1. ↗
  28. Haoan Jin, Jiacheng Shi, Hanhui Xu, Kenny Q. Zhu, and Mengyue Wu. 2025. Medethiceval: Evaluating large language models based on chinese medical ethics. 2503.02374v1. 26479 11 find
  29. Zahra Khalila, Arbi Haza Nasution, Winda Monika, Aytug Onan, Yohei Murakami, Yasir Bin Ismail Radi, and Noor Mohammad Osmani. 2025. Investigating retrieval-augmented generation in quranic studies: A study of 13 open-source large language models. 2503.16581v1. International Journal of Advanced Computer Science and Applications(IJACSA), 16(2), 2025. ↗
  30. Abderraouf Lahmar, Md Easin Arafat, Zakarya Farou, and Mufti Mahmud. 2025. Islamtrust: A benchmark for llms alignment with islamic values. In Proceedings of the 5th Muslims in ML Workshop at NeurIPS 2025. find
  31. Yangning Li, Weizhi Zhang, Yuyao Yang, Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, Chunkit Chan, Yankai Chen, Zhongfen Deng, Yinghui Li, Hai-Tao Zheng, Dongyuan Li, Renhe Jiang, Ming Zhang, Yangqiu Song, and Philip S. Yu. 2025. Towards agentic rag with deep reasoning: A survey of rag-reasoning systems in llms. 2507.09477v2. ↗
  32. Jintao Liang, Gang Su, Huifeng Lin, You Wu, Rui Zhao, and Ziyue Li. 2025. Reasoning rag via system 1 or system 2: A survey on reasoning agentic retrieval-augmented generation for industry challenges. 2506.10408v1. ↗
  33. Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, and Jinchao Xu. 2024. Alignment at pre-training! towards native alignment for arabic llms. Preprint, arXiv:2412.03253. ↗
  34. Rana Malhas and Tamer Elsayed. 2020. Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  35. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. Qur’an qa 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection. Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno M Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M Alves, José Pombal, Nicolas Boizard, and 1 others. 2025. Eurollm-9b: Technical report. arXiv preprint arXiv:2506.04079. ↗
  36. Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Mohamed Darwish, and Walid Magdy. 2025. IslamicEval 2025: The first shared task of capturing LLMs hallucination in islamic content. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 480–493, Suzhou, China. Association for Computational Linguistics.
  37. Tarek Naous and Wei Xu. 2025. On the origin of cultural biases in language models: From pre-training data to linguistic phenomena. 2501.04662v1. ↗
  38. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b and gpt-oss-20b model card. Preprint, arXiv:2508.10925. ↗
  39. Raghad Salameh, Mohamad Al Mdfaa, Nursultan Askarbekuly, and Manuel Mazzara. 2024. Quranic audio dataset: Crowdsourced and labeled recitation from non-arabic speakers. Procedia Computer Science, 246:2684–2693.
  40. Peizhang Shao, Linrui Xu, Jinxi Wang, Wei Zhou, and Xingyu Wu. 2025. When large language models meet law: Dual-lens taxonomy, technical advances, and ethical governance. 2507.07748v1. ↗
  41. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300. silma-ai. 2024. Silma 9b instruct https://huggingface.co/silma-ai/ SILMA-9B-Instruct-v1.0. v1.0. ↗
  42. Jannatul Tajrin, Bir Ballav Roy, and Firoj Alam. 2025. AYA at PalmX 2025: Modeling cultural and islamic knowledge in LLMs. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 830–836, Suzhou, China. Association for Computational Linguistics.
  43. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv: 2501.13944. ↗
  44. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. ↗
  45. Saad Obaid ul Islam, Anne Lauscher, and Goran Glavaš. 2025. How much do llms hallucinate across languages? on multilingual estimation of llm hallucination in the wild. 2502.12769v3. find
  46. Changyue Wang, Weihang Su, Qingyao Ai, and Yiqun Liu. 2025. Joint evaluation of answer and reasoning consistency for hallucination detection in large reasoning models. 2506.04832v1. ↗
  47. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. 26480 12 ↗
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  49. Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and Lidong Bing. 2025. SeaLLMs 3: Open foundation and chat multilingual large language models for Southeast Asian languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pages 96–105, Albuquerque, New Mexico. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhatia-2026-rag-agentic-rag-faithful-islamic/