📄 Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering

Summary

This paper describes a system entered in Task 2.1 of HalluScoring 2026, a shared task on spotting hallucinations in Islamic content and identifying the truthful answer. The task asks for two linked decisions about an Arabic answer that an LLM wrote for an Islamic question. First, the system labels the answer as hallucinated or not. Second, it picks the verified answer from six very similar candidate options.

The author fine-tunes google/gemma-4-12B-it with a separate adapter for each subtask. Decoding is deterministic, and a normalization step extracts the label and the chosen option from the generated text. Training data come from several Arabic resources: a 43,400-pair detection corpus and a 2,600-question answer-selection set.

On the official 600-instance evaluation set, detection reaches Macro-F1 0.928 and option selection reaches 0.895 accuracy, for a combined score of 0.912. The error analysis is the most useful part. It shows that recognizing a bad answer does not mean the model knows the correct one: the model sometimes rejects a response and then chooses the option that repeats the same mistake. This matters in domains where fabricated religious evidence can look authoritative.

Key points

• The evaluation data have 600 Arabic QA instances: 341 hallucinated (56.8%) and 259 non-hallucinated, with 393 manually annotated hallucination spans.
• The detection corpus mixes several sources: IslamWeb fatwas, QIAS 2025 MCQs, MAWARITH inheritance problems, PalmX 2025, IslamicFaithQA, HalluTruthQA, 5,000 IslamQA items with deliberately edited Qur'anic/hadith quotations, and 3,000 Gemini-generated general-knowledge answers. The corpus was balanced between the two labels.
• The answer-selection set reuses 2,000 QIAS 2025 questions and 600 six-option HalluTruthQA questions, keeping their original near-miss distractors.
• Results: Macro-F1 0.928, label accuracy 0.935, option accuracy 0.895. The final score of 0.912 is an equal-weight average of Macro-F1 and option accuracy.
• The error analysis identifies three failure types:
  • apparent false negatives, where a different verse also answers the question but disagrees with the single reference;
  • a correct "hallucinated" verdict followed by selection of the same wrong answer (Anas ibn Mālik instead of Abū al-Tufayl);
  • confusion between closely related entities.

Interesting ideas

• I1. MCQ items become binary detection examples by pairing each question with either its gold option or one of its own human-written distractors, which gives topical, fluent negatives for free.
• I2. Small controlled edits (substituting, deleting or inserting a word inside a quotation) produce localized errors while the surrounding text stays fluent and correct.
• I3. Detecting a wrong answer and recovering the right one are scored as separate capabilities, and the paper proposes a consistency rule: once an answer is rejected, an option repeating its central claim should be penalized.
• I4. The paper reports an anchoring bias: selection tends to favor the option with the most lexical overlap with a previously seen answer.
• I5. Single-reference scoring can mark valid alternative answers as errors, so the paper suggests multiple references or more specific questions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova could build hard negative controls from existing multiple-choice school tests. It would pair a question with a wrong option and check that the reader rejects it instead of matching on topic.
• I2 (answer): Minimally perturbed versions of fable or word-problem texts, such as a changed number or swapped name, would test whether the answer truly comes from the world of the text.
• I3 (answer): Mova's readers could run an explicit check that a claim judged false in the world model is never returned as the answer. Each violation would be logged as an explainable inconsistency.
• I4 (learn): Evaluation could add a control in which a distractor copies wording from the question or text, measuring whether answers rely on surface overlap rather than frames and entities.
• I5 (learn): Mova's QA benchmarks could allow sets of acceptable answers, or flag underspecified questions, so that correct but different readings are not counted as failures.