Elamur Research⌕
☰
home / papers / 2025 / BurhanAI at IslamicEval 2025 Shared Task: Combating Hallucinations in LLMs for Islamic Content; Evaluation, Correction, and Retrieval-Based Solution

BurhanAI at IslamicEval 2025 Shared Task: Combating Hallucinations in LLMs for Islamic Content; Evaluation, Correction, and Retrieval-Based Solution

Paper · 2025 · Arij Al Adel, Abu Bakr Soliman, Mohamed Sakher Sawan et al.

Summary

The paper presents the BurhanAI team's entry to IslamicEval 2025, which targets LLM hallucinations about Islamic texts: invented Hadiths, misquoted Quranic verses and similar errors. The team took part in four subtasks:

  • 1A: locating Ayah/Hadith spans inside model answers.
  • 1B: judging whether a span is authentic.
  • 1C: replacing wrong spans with the canonical text.
  • 2: retrieving answer-bearing passages from the Quran and Sahih Al-Bukhari.

For 1A, they fine-tuned gpt-4.1-mini on a mix of shared-task data and gpt-4.1-generated synthetic examples. They compared it with reasoning models that call a code interpreter to compute exact character offsets. For 1B and 1C, they built a multi-index lookup that escalates from exact hashing to fuzzy and semantic matching, with a cheap LLM as the final judge. For subtask 2, GPT-5 ran a corpus-restricted RAG loop.

Main scores: 90.06% F1 on 1A with o4-mini plus tools, 88.60% accuracy on validation and 66.56% on correction. The broader lesson is that careful normalization, closed-world search and small or fine-tuned models can match or beat costly frontier systems on narrow tasks.

Key points

  • Subtask 1A (macro F1): o4-mini (high reasoning) with code interpreter 90.06%; o3-mini 87.78%; fine-tuned gpt-4.1-mini 87.10%. The fine-tuned model was trained on 460 training and 83 validation examples, about 70% real and 30% synthetic, for 3 epochs. Larger models (o3, all GPT-5 sizes) did not reach the top three.
  • Indexing for 1B/1C: seven index types are used: MD5 of raw text, diacritic- and punctuation-free normalized text, character 3-grams, length buckets, an edit-distance list, Whoosh full-text, and Cohere embed-v4.0 vectors in Qdrant.
  • Search for 1B/1C: a six-stage search with early stopping, plus Cohere rerank-v3.5 and FuzzyWuzzy. Results were 88.60% on 1B and 66.56% on 1C, beating GPT-5 with tools on 1C (61.04%).
  • Subtask 2: RAG over 1,500-token chunks with 400-token overlap, with GPT-5 reformulating queries over several iterations. Ayat were mapped to QPC passage IDs and hadith IDs were checked against the official JSONL. MAP@10 was 0.6199 on dev and 0.2807 on test (MAP_Q@5 0.3257, MAP_H@5 0.2386).
  • 1A post-processing: outputs follow a JSON schema; labels are refined using cues within ±64 characters; punctuation is trimmed; nested spans are merged or split.

Interesting ideas

  • I1. Matching runs as a strict cascade from cheapest and most exact (hash, normalized form) to loosest (n-grams, embeddings, edit distance, token overlap, trigram Jaccard), stopping at the first confident hit, so clean inputs cost almost nothing while noisy ones still get recovered.
  • I2. Instead of trusting an LLM to count characters, the model writes code that computes the offsets, and the numbers come from deterministic execution.
  • I3. Free-form generation is replaced by a fixed JSON schema, and a deterministic pass then snaps outputs to canonical IDs and drops duplicates or invalid references.
  • I4. After a span is found, a small window of surrounding characters is inspected for lexical cues (e.g., introductory formulae) to fix or confirm the span's label.
  • I5. A small model fine-tuned on mostly real plus some synthetic data matched much larger models on a narrow task.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Linking text mentions to WordNet/Wikidata entries could use the same staged lookup, recording which stage fired so every link carries an explainable confidence reason.
  • I2 (learn): When LLM teacher tiers propose spans or token alignments, offsets should be recomputed and checked by code before the labels enter silver data.
  • I3 (answer): Readers answering QA on fables could emit schema-bound answers that must resolve to node IDs in the world graph, rejecting any answer that points to nothing.
  • I4 (frames): Detecting reported speech and quotations could check nearby cue verbs and markers to decide the speech-act frame and attribution of the quoted span.
  • I5 (learn): Small specialized students trained on mixed silver and synthetic examples are worth benchmarking against frontier teachers before assuming the bigger model is needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Arij Al Adel, Abu Bakr Soliman, Mohamed Sakher Sawan, Rahaf Al-Najjar and Sameh Amin. 2025. BurhanAI at IslamicEval 2025 Shared Task: Combating Hallucinations in LLMs for Islamic Content; Evaluation, Correction, and Retrieval-Based Solution. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.69

@inproceedings{adel2025burhanai,
  title = {{BurhanAI at IslamicEval 2025 Shared Task: Combating Hallucinations in LLMs for Islamic Content; Evaluation, Correction, and Retrieval-Based Solution}},
  author = {Arij {Al Adel}, Abu Bakr Soliman, Mohamed Sakher Sawan, Rahaf Al-Najjar and Sameh Amin},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.69},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.69}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydin. 2024. A rag-based question answering system proposal for understanding islam: Mufassirqas llm. ArXiv, abs/2401.15378. ↗
  2. Liam Barkley and Brink van der Merwe. 2024. Investigating the role of prompting and external tools in hallucination rates of large language models. ArXiv, abs/2410.19385. ↗
  3. Patrice B’echard and Orlando Marquez Ayala. 2024. Reducing hallucination in structured outputs via retrieval-augmented generation. In North American Chapter of the Association for Computational Linguistics. find
  4. Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, and Jiahui Wen. 2025. Think more, hallucinate less: Mitigating hallucinations via dual process of fast and slow thinking. ArXiv, abs/2501.01306. ↗
  5. Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. 2024. Why do large language models (llms) struggle to count letters? arXiv preprint arXiv:2412.18626. ↗
  6. Amina El Ganadi, Sania Aftar, Luca Gagliardelli, and Federico Ruozzi. 2025. Generative ai for islamic texts: The eman framework for mitigating gpt hallucinations. In International Conference on Agents and Artificial Intelligence. find
  7. Baraa Hikal, Ahmed Nasreldin, and Ali Hamdi. 2025. Msa at semeval-2025 task 3: High quality weak labeling and llm ensemble verification for multilingual hallucination detection. ArXiv, abs/2505.20880. ↗
  8. Ben Hutchinson. 2024. Modeling the sacred: Considerations when using considerations when using religious texts in natural language processing. In NAACL-HLT. find
  9. Zahra Khalila, Arbi Haza Nasution, Winda Monika, Aytuğ Onan, Yohei Murakami, Yasir Bin Ismail Radi, and Noor Mohammad Osmani. 2025. Investigating retrieval-augmented generation in quranic studies: A study of 13 open-source large language models. ArXiv, abs/2503.16581. ↗
  10. Johnny Li, Saksham Consul, Eda Zhou, James Wong, Naila Farooqui, Yuxin Ye, Nithyashree Manohar, Zhuxiaona Wei, Tian Wu, Ben Echols, Sharon Zhou, and Gregory Diamos. 2024. Banishing llm hallucinations requires rethinking generalization. ArXiv, abs/2406.17642. ↗
  11. Shuyuan Lin, Lei Duan, Philip Hughes, and Yuxuan Sheng. 2025. Harnessing rlhf for robust unanswerability recognition and trustworthy response generation in llms. arXiv preprint arXiv:2507.16951. ↗
  12. Rana Malhas and Tamer Elsayed. 2020. Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. *Figure 1: The plot generated from the fine-tuning loss table provided by the OpenAI platform.* A.1.2 find
  13. Marryam Yahya Mohammed, Sama Ayman Ali, Salma Khaled Ali, Ayad Abdul Majeed, and Ensaf Hussein Mohamed. 2025. Aftina: enhancing stability and preventing hallucination in ai-based islamic fatwa generation using llms and rag. Neural Computing and Applications. Model Name GPT-5 Nano O3 GPT-5 GPT-5 Nano GPT-5 Nano O4 Mini GPT-5 O3 Mini GPT-5 GPT-5 Mini GPT-5 GPT-5 Mini GPT-5 Mini Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Darwish, and Walid Magdy. 2025. IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. ↗
  14. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2024. Llms know more than they show: On the intrinsic representation of llm hallucinations. ArXiv, abs/2410.02707. ↗
  15. Serry Sibaee, Abdullah I. Alharbi, Samar Ahmed, Omar Nacar, Lahouri Ghouti, and Anis Koubaa. 2024. Asos at arabic llms hallucinations 2024: Can llms detect their hallucinations :). In OSACT.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adel-2025-burhanai-islamiceval-2025-shared-task/