📄 HUMAIN at IslamicEval 2025 Shared Task 1: A Three-Stage LLM-Based Pipeline for Detecting and Correcting Hallucinations in Quran and Hadith

Summary

The paper tackles a problem specific to religious content: LLMs often produce Quranic verses or Hadith that look plausible but are misquoted, misattributed or invented. HUMAIN entered all three parts of IslamicEval 2025 Subtask 1, which asks systems to find such quotations in Arabic LLM output, decide whether each one is authentic, and replace faulty ones with the genuine reference text.

Their pipeline has three stages:

• Detection. An LLM is prompted, without fine-tuning, to rewrite the input with inline span markup in the TANL style. The markup is then aligned back to the original text.
• Validation. Normalized spans are checked against the Quran and Hadith collections, using lookup indices and either fuzzy LCS matching or strict substring matching depending on the source.
• Correction. Candidates pass through a cascade of exact matching, LCS similarity and a multilingual semantic reranker.

On the test set the system reached 87.2% F1 for detection, 86.1% accuracy for validation and 68.2% for correction. The authors conclude that finding quotations is largely solved, but fixing them is still hard. Corrections are most difficult when a hallucinated span shares its theme with the true text but uses very different wording.

Key points

• Data. All data is Arabic. Each subtask has 50 dev and 104 test samples, scored on CodaBench. Validation and correction use the Quran and six Hadith books as reference corpora.
• Detection models. The authors compared GPT-4o, ALLAM-34B, Command-R7B-Arabic, Fanar and Jais-13B, all prompted with few-shot examples and none fine-tuned.
  • Dev F1: GPT-4o 81.1, ALLAM-34B 77.2, Command-R7B-Arabic 48.1, Fanar 27.0, Jais-13B 12.9.
  • Test F1: GPT-4o 87.2, ALLAM-34B 78.1.
• Best validation setting. Quran spans used LCS-based fuzzy matching with a 0.90 threshold, and Hadith spans used strict substring matching, giving 86.14% test accuracy. The authors explain the split by the corpora themselves: Quran text circulates in several valid script forms, such as Uthmani or with and without diacritics, while Hadith collections are formatted consistently.
• Correction cascade. Candidates are first pre-filtered with a word inverted index plus a character 3-gram index. Up to 20 candidates are then reranked with bge-reranker-v2-m3, whose score is mixed with LCS similarity using weight α = 0.7. Thresholds are 0.85 for Quran and 0.75 for Hadith, and spans below threshold are labelled W (no valid correction).
• Correction results. Test accuracy rose from 57.48% with LCS alone, to 65.91% with exact matching plus LCS, to 68.18% with the reranker added. Code and prompts are released on GitHub.

Interesting ideas

• I1. Let the LLM reproduce the passage with inline tags, then recover exact source offsets with Needleman–Wunsch alignment, so the model's small edits to spacing or wording do not break the annotation
• I2. Use schema-constrained JSON decoding (via vLLM) when the model is hosted locally, and fall back to markup plus alignment for API-only models
• I3. Pick the matching strictness per reference source: fuzzy where several legitimate variants of a text exist, strict where the text is standardized
• I4. Run a cheap-to-expensive cascade of exact match, then LCS, then a semantic reranker on the top candidates, blend lexical and semantic scores, and abstain with an explicit "uncorrectable" label below threshold
• I5. Before checking a span, detect whether it actually contains several quoted units (from separators and verse numbers), verify each part separately, and combine the verdicts

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): LLM teacher tiers could return tagged text that is aligned back onto our own tokens, turning messy teacher output into usable silver labels.
• I2 (learn): keeping both output modes behind one interface would let us swap teacher models without rewriting how their annotations are parsed.
• I3 (world): when linking entity mentions to WordNet/Wikidata labels, we could tune tolerance per resource instead of using one global similarity cutoff.
• I4 (answer): our readers could match answer options or retrieve supporting facts this way, with each step logged for explainability and an explicit "no answer" outcome instead of a weak guess.
• I5 (frames): splitting compound quotations or reported speech into units before attributing them would make speech-act and source tracking more precise.