Elamur Research⌕
☰
home / papers / 2025 / Isnad AI at IslamicEval 2025: A Rule-Based System for Identifying Religious Texts in LLM Outputs

Isnad AI at IslamicEval 2025: A Rule-Based System for Identifying Religious Texts in LLM Outputs

Paper · 2025 · Mohamed Emad Elden, Fatimah

Summary

Isnad AI is a system for IslamicEval 2025 Shared Task 1A. The task is to find the exact character positions where an Arabic LLM answer quotes a Quranic verse (Ayah) or a Prophetic saying (Hadith). The authors treat this as token classification with BIO labels and fine-tune AraBERTv2. No large manually labeled corpus exists for this task, so their main contribution is a rule-based pipeline that builds training data from the raw religious texts supplied by the organizers. The pipeline splits long verses, adds copies without diacritics, and places each authentic text inside hand-curated prefix and suffix templates that mimic how LLMs introduce citations. The model trained on this synthetic corpus reached a character-level macro F1 of 66.97% on the blind test set. It beat a database-lookup baseline and a plain fine-tuning baseline. Hadith detection was the weakest part. The work shows that carefully controlled synthetic data can replace scarce annotation in a narrow verification task, and the code, data and model are released.

Key points

  • Sources: 6,236 Ayahs and 34,662 Hadiths from the Six Major Books, keeping only entries with a non-empty Matn. Verses longer than 25 AraBERTv2 tokens were split in two, which raised the Ayah count to 6,910. This yielded 43,902 unique texts and 133,725 generated examples.
  • The train/validation split was 70/30 over unique source texts. Official evaluation used 210 dev records and 190 blind test records.
  • The official metric is character-level macro F1 over the labels Ayah, Hadith and Neither:
    • The rule-based model scored 65.08% on dev and 66.97% on test.
    • Database lookup scored 52% on dev and 34.80% on test.
    • Basic fine-tuning scored 33% on dev and 44.70% on test.
  • Class imbalance was strong. Neither covered 67.8% of characters, Ayah 20.2% and Hadith 12.0%. Per-class F1 was 0.90, 0.67 and 0.39 respectively.
  • At span level the model missed 101 spans and got 78 right, with missed spans being whole citations rather than partial ones. False positives were mostly short fragments. Synthetic data generated with AraGPT2 proved too noisy to use, because it contained hallucinations and nonsense.

Interesting ideas

  • I1. The gold spans are known by construction because real target texts are inserted into curated context templates, so no annotation is needed to get labeled boundaries.
  • I2. The data is split by unique source text before any template expansion, so the same verse never appears in both training and validation inside different wrappers.
  • I3. Each text gets a duplicate with diacritics stripped, which teaches the model to cope with surface-form variation of the same content.
  • I4. The authors tested LLM-generated synthetic contexts against rule-based templates and rejected the LLM data because its hallucinations corrupted the labels.
  • I5. Beyond the character-level score, a span-level error analysis separates entirely missed citations from short spurious fragments, which point to different fixes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): We could build silver data for quoted or reported speech by inserting known sentences into "he said that…" style frames, so span and role labels are exact by design.
  • I2 (learn): When we expand sentences into augmented variants, the train/eval split should happen at the source-sentence level so that negative controls are not leaked.
  • I3 (tokens): An English counterpart would add lowercased, punctuation-light or contraction-expanded copies to tagger and lemma training, making them robust to messy input.
  • I4 (learn): Outputs from the LLM teacher tiers should pass a structural check, or be compared with rule-made data, before they enter training.
  • I5 (answer): Our readers could report missed answer spans separately from spurious short spans, which would show whether a failure comes from recall or from over-triggering.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Mohamed Emad Elden, Fatimah. 2025. Isnad AI at IslamicEval 2025: A Rule-Based System for Identifying Religious Texts in LLM Outputs. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.74

@inproceedings{fatimah2025isnad,
  title = {{Isnad AI at IslamicEval 2025: A Rule-Based System for Identifying Religious Texts in LLM Outputs}},
  author = {Mohamed Emad Elden, Fatimah},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.74},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.74}
}

References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2025. Al-Qur’an al-Karim (Mushaf al-Madinah an-Nabawiyyah). King Fahd Complex for the Printing of the Holy Qur’an, Medina, Saudi Arabia. Arabic text based on the Uthmanic (script), in the narration of Hafs from Asim. Corresponds to Hijri year 1446-1447 AH. find
  2. Shared Task of Capturing LLMs Hallucination in Islamic Content. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. find
  3. Muhammad Ismail al Bukhari. 1871. Sahih al-Bukhari. King Fahd National Library - Riyadh. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196– 207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Kais Dukes and Nizar Habash. 2010. Morphological annotation of Quranic Arabic. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA).
  8. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  9. Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora.
  10. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. A Experimental Configuration Table 3 provides the hyperparameter settings used for fine-tuning the AraBERTv2 model (Antoun et al., 2020). ↗
  11. Ellbendis. 2024. Qwen3-4b-quran-lora-fine-https://huggingface.co/Ellbendls/ tuned. Qwen3-4b-Quran-LoRA-Fine-Tuned. Accessed: 2025-08-16. Value Model aubmindlab/bert-base-arabertv2 10 (with early stopping) 2 × 10 −5 4 4 Max Epochs Learning Rate Batch Size (per device) Gradient Accumulation Steps Effective Batch Size Optimizer Weight Decay Warmup Steps Mixed Precision Max Sequence Length Early Stopping Patience Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568, Online. Association for Computational Linguistics. ↗
  12. Justdeen. 2024. Quranplus. https://huggingface.co/ justdeen/QuranPlus. Accessed: 2025-08-16. Parameter 16 AdamW 0.01 500 fp16 enabled 512 tokens 3 epochs *Table 3: Complete hyperparameter configuration for model training.* Siheng Li, Cheng Yang, Taiqiang Wu, Chufan Shi, Yuji Zhang, Xinyu Zhu, Zesen Cheng, Deng Cai, Mo Yu, Lemao Liu, Jie Zhou, Yujiu Yang, Ngai Wong, Xixin Wu, and Wai Lam. 2024. A survey on the honesty of large language models. arXiv preprint arXiv:2409.18786. ↗
  13. Hamdy Mubarak, Rana Malhas, Watheq Mansour, Abubakr Mohamed, Mahmoud Fawzi, Majd Hawasly, Tamer Elsayed, Kareem Darwish, and Walid Magdy. 2025. IslamicEval 2025: The First Question_ID Span_Start Span_End Span_Type A-Q001 A-Q001 A-Q001 11 34 67 25 52 87 Ayah Ayah Hadith *Table 4: Example submission file format.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elden-2025-isnad-ai-islamiceval-2025-rule/