Elamur Research⌕
☰
home / papers / 2025 / IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content

IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content

Paper · 2025 · Hamdy Mubarak, Rana Malhas, Watheq Mansour et al.

Summary

LLMs used for Arabic religious questions often misquote or invent Qur'anic verses and Hadiths. IslamicEval 2025 is the first shared task built around measuring and catching such errors. It has two parts.

The first part breaks hallucination handling into three steps:

  • locating the text an answer intends as a verse or Hadith,
  • judging whether that quote is authentic,
  • supplying the exact canonical wording.

Its data comes from questions in the Qur'an QA 2023 dataset. Six Arabic-centric and multilingual LLMs were asked to answer them with cited evidence. Islamic-studies experts then labeled the outputs in Label Studio and corrected the faulty quotes against the Uthmani Qur'an and the six canonical Hadith collections.

The second part is a retrieval task. Given a Modern Standard Arabic question, systems return up to 20 ranked answer-bearing passages drawn from both the Qur'an and Sahih al-Bukhari.

Thirteen teams reached the final phase. The resource matters because generated religious text is usually not checked against authoritative sources, and this task gives a way to do that.

Key points

  • The corpus has 1,506 annotated answers (251 questions × 6 LLMs). The released dev set has 150 answers and the test set 312; a revised training split of 1,044 answers is planned.
  • Hallucination rates differ widely by model and by source type:
    • ALLaM-7B-Instruct-preview quoted 1,104 verses (84.06% correct) and 654 Hadiths (59.33% correct).
    • An in-house fine-tuned Gemma-2-9b reached only 65.33% correct verses and 38.17% correct Hadiths.
  • Each subtask has its own metric:
    • Subtask A (spans): character-level macro F1 over three classes (verse, Hadith, neither).
    • Subtask B (correct/incorrect): accuracy.
    • Subtask C (correction): strict exact-match accuracy after stripping default diacritics such as sukun.
  • The task ran as three CodaBench competitions, with participants limited to the provided data. Subtask 1A drew 20 participants (87 submissions) and 1B drew 16 (41 submissions).
  • Subtask 2 teams (Burhan, ThinkDrill, NUR, BurhanAI) relied on several techniques:
    • LLM-based data augmentation,
    • LLM or cross-encoder reranking,
    • fine-tuned or carefully selected embedding models,
    • question paraphrasing,
    • explicit detection of questions with no answer.

Interesting ideas

  • I1. Annotators mark the span a model meant as a quote even when its wording is wrong, so finding a claim is scored separately from checking it.
  • I2. An incorrect item gets either its nearest true version or an explicit "no such thing exists" label, never a forced guess.
  • I3. Span detection is scored per character with macro averaging across classes, so boundary slips cost partial credit and the large "neither" class cannot hide misses.
  • I4. Corrections must match the reference exactly, but only after a fixed normalization removes differences that do not change meaning (default diacritics).
  • I5. Several teams treated "no passage answers this" as a first-class outcome, using score thresholds, an LLM yes/no judge, or negative training samples.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Statements in a text could first be anchored as intended claims and then tested against the knowledge graph, with failures traced to "located" or "verified" separately.
  • I2 (world): When a stated fact conflicts with compiled knowledge, the world model could record either the closest true entry or a "no counterpart" state instead of silently patching it.
  • I3 (learn): The same per-character macro F1 could score quotation or named-span detection in Mova's evaluations, where target spans are rare compared with background text.
  • I4 (back): Round-trip checks of the text → graph → text compressor could use strict match after a declared normalization step, so formatting noise is ignored while real losses are caught.
  • I5 (answer): Readers answering school tests or fable QA could abstain explicitly when no frame in the text world supports an answer, and be tested on deliberately unanswerable questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓The University of QueenslandBrisbane · Australia14 works · 4,603 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Mohamed Darwish and Walid Magdy. 2025. IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.67

@inproceedings{mubarak2025islamiceval,
  title = {{IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content}},
  author = {Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Mohamed Darwish and Walid Magdy},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.67},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.67}
}

References

45 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin. 2025. Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations. arXiv preprint arXiv:2503.07833. ↗
  2. Norah Abokhodair, AbdelRahim Elmadany, and Walid Magdy. 2020. Holy tweets: Exploring the sharing of the quran on twitter. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2):1–32. find
  3. Basem H. Ahmed, Motaz K. Saad, and Eshrag A. Refaee. 2022. QQATeam at Quran QA 2022: Fine-Tuning Arabic QA Models for Quran QA Task. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022). find
  4. Arij Al Adel, Abu Bakr Soliman, Mohamed Sakher Sawan, Rahaf Al-Najjar, and Sameh Amin. 2025. Combating hallucinations in llms for islamic content: Evaluation, correction, and retrieval-based solution. In Proceedings of the Third Arabic Natural Language Processing Conference. find
  5. Hend Al-Khalifa, Tamer Elsayed, Hamdy Mubarak, Abdulmohsen Al-Thubaity, Walid Magdy, and Kareem Darwish. 2022. Proceedinsg of the 5th workshop on osact with shared tasks on qur’an qa and fine-grained hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection. find
  6. Z. A. B. Al-Zubaidi. 2009. Al-Tajreed Al-Sareeh of Collective Sahih Hadith. Resalah Publishers. Author died 893 AH/1488 CE. find
  7. Hayfa A Aleid and Aqil M Azmi. 2025. Hajj-fqa: A benchmark arabic dataset for developing question-answering systems on hajj fatwas: H. aleid and a. azmi. Journal of King Saud University Computer and Information Sciences, 37(6):135. ↗
  8. Serag Amin, Ranwa Aly, Yara Allam, Yomna Eid, and Ensaf Hussein Mohamed. 2025. Nur at islamiceval 2025 shared task: Retrieval-augmented llms for qur’an and hadith qa. In Proceedings of the Third Arabic Natural Language Processing Conference.
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Araelectra: Pre-training text discriminators for arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195. Association for Computational Linguistics. ↗
  11. Aqil M Azmi, Abdulaziz O Al-Qabbany, and Amir Hussain. 2019. Computational and natural language processing based studies of hadith literature: a survey. Artificial Intelligence Review, 52:1369–1414. ↗
  12. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. ALLam: Large language models for Arabic and English. arXiv preprint arXiv:2407.15390. ↗
  13. Mohammad Basheer, Watheq Mansour, Abdulhamid Touma, and Ahmad Qadeib Alban. 2025. Burhan at islamiceval: Fact-augmented and llm-driven retrieval for islamic qa. In Proceedings of the Third Arabic Natural Language Processing Conference.
  14. Amina El Ganadi, Sania Aftar, Luca Gagliardelli, Federico Ruozzi, et al. 2025. Generative ai for islamic texts: The eman framework for mitigating gpt hallucinations. In roceedings of the 17th International Conference on Agents and Artificial Intelligence-ICAART, volume 3, pages 1221–1228. find
  15. Fatimah Mohamed Emad Elden. 2025. Isnad ai at islamiceval 2025: A rule-based system for identifying religious texts in llm outputs. In Proceedings of the Third Arabic Natural Language Processing Conference.
  16. Mohammed ElKomy and Amany M. Sarhan. 2022. TCE at Qur’an QA 2022: Arabic Language Question Answering Over Holy Qur’an Using a Post-Processed Ensemble of BERT-based Models. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022). ↗
  17. Mohammed Alaa Elkomy and Amany Sarhan. 2023. TCE at Qur’an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur’anic QA. In Proceedings of the First Arabic Natural Language Processing Conference (Arabic-NLP 2023), Singapore.
  18. Mohammed ElKoumy, Mohamed Ibrahim Alqablawi, Ahmad Tamer, and Khalid Allam. 2025. Tce at islamiceval 2025: Retrieval-augmented llms for quranic and hadith content identification and verification. In Proceedings of the Third Arabic Natural Language Processing Conference.
  19. Eman Elrefai, Toka Khaled, and Ahmed Soliman. 2025. Thinkdrill at islamiceval 2025: Llm hybrid approach for quran and hadith question answering. In Proceedings of the Third Arabic Natural Language Processing Conference. find
  20. Fanar Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. ↗
  21. Mahmoud Fawzi, Walid Magdy, and Björn Ross. 2025. ’the prophet said so!’: On exploring hadith presence on arabic social media. Proceedings of the ACM on Human-Computer Interaction, 9(2):1–23. ↗
  22. Mahmoud Fawzi, Björn Ross, and Walid Magdy. 2026. Fabricating holiness: Characterizing religious misinformation circulators on arabic social media. Proceedings of the International AAAI Conference on Web and Social Media, 20. find
  23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and Arun Rao et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  24. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38. ↗
  25. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474.
  26. Jimmy Lin and Boris Katz. 2006. Building a reusable test collection for question answering. Journal of the American Society for Information Science and Technology, 57(7):851–861. ↗
  27. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021. Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies, 14(4):1–325. ↗
  28. Rana Malhas and Tamer Elsayed. 2020. AyaTEC: Building a Reusable Verse-based Test Collection for Arabic Question Answering on the Holy Qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing, 19(6):1–21. find
  29. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  30. Qur’an QA 2022: Overview of the First Shared Task on Question Answering over the Holy Qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022), pages 79–87.
  31. Marwan N. Swar. 2007. Mushaf Al-Tafseel Al-Mawdoo’ee. Dar Al-Fajr Al-Islami, Damascus. find
  32. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. Qur‘an QA 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur‘an. In Proceedings of ArabicNLP 2023, pages 690–701, Singapore (Hybrid). Association for Computational Linguistics.
  33. Raúl Vázquez, Timothee Mickus, Elaine Zosa, Teemu Vahtola, Jörg Tiedemann, Aman Sinha, Vincent Segonne, Fernando Sánchez-Vega, Alessandro Raganato, Jindřich Libovickỳ, et al. 2025. Semeval-2025 task 3: Mu-shroom, the multilingual shared task on hallucinations and related observable overgeneration mistakes. arXiv preprint arXiv:2504.11975. ↗
  34. Rana R Malhas. 2023. ARABIC QUESTION ANSWER-ING ON THE HOLY QUR’AN. Ph.D. thesis, Qatar University. find
  35. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. ↗
  36. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data labeling software. Open source software available from https://github.com/HumanSignal/label-studio. ↗
  37. Abdulrezzak Zekiye and Fadi Amroush. 2023. Al-jawaab at Qur’an QA 2023 shared task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  38. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. Fine-grained hallucination detection and editing for language models. arXiv preprint arXiv:2401.06855. ↗
  39. Hamdy Mubarak, Hend Al-Khalifa, and Khaloud Suliman Alkhalefah. 2024. Halwasa: Quantify and analyze hallucinations in large language models: Arabic as a case study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8008–8015.
  40. Al-Tahir A.R. Musallam. 2022. Prophetic interpretation of the quran: Between quantity and quality. Rehan Journal for Scientific Publishing, 26(1):51–76. find
  41. Arwa Omayrah, Sakhar Alkhereyf, Ahmed Abdelali, Abdulmohsen Al-Thubaity, Jeril Kuriakose, and Ibrahim AbdulMajeed. 2025. Humain at islamiceval 2025 shared task 1: A three-stage llm-based pipeline for detecting and correcting hallucinations in quran and hadith. In Proceedings of the Third Arabic Natural Language Processing Conference.
  42. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. arXiv preprint arXiv:2101.05779. ↗
  43. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  44. Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922. ↗
  45. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mubarak-2025-islamiceval-2025-first-shared-task/