📄 A Knowledge Graph Based Diagnostic Framework for Analyzing Hallucinations in Arabic Machine Reading Comprehension

Summary

LLMs answering questions about a given passage often produce fluent statements that the passage does not actually support. Little of this has been studied for Arabic reading comprehension, and almost none of it for Qur'anic texts, where Classical Arabic, implicit meaning and interpretive sensitivity make errors costly. The authors do not build a new detector or metric. Instead they offer a diagnostic framework that works entirely in Arabic.

The framework turns the passage, the question and the answer into subject–relation–object triples and compares them. A question-aware filter decides which content is relevant to what was asked. Supervision is weak: only the passage, question and gold answer are assumed, with no hallucination labels, and humans step in only for ambiguous triples.

Applied to model answers on QRCD, the framework shows systematic error patterns. Questions asking for evidence, justification or abstract interpretation trigger far more problem triples than factual ones. This is a contrast that surface similarity scores hide. The work argues for transparent, structured evaluation in low-resource, high-stakes settings.

Key points

• Data: the Qur'anic Reading Comprehension Dataset (QRCD) has 992 passage–question–answer instances over 603 unique passages (about 1.64 questions per passage). Each passage is typically two to five verses, and each instance has a human-annotated gold answer.
• Each triple receives one of five labels: supported, missing, hallucinated, off-focus, or gold-only (found in the gold answer but not recoverable from the passage).
• The framework extracted 23,388 triples from gold and model answers. Of these, 6,233 were flagged for review because of uncertain question relevance or conflicts between passage support and gold annotations.
• Measured against passage-derived reference graphs, triple extraction reached 74.6% precision and 57.7% recall. This limits how far the downstream diagnosis can be trusted.
• Typical failures:
  • hallucinations were plausible generic religious claims absent from the passage;
  • missing triples were often secondary entities or implicitly expressed relations;
  • many gold-only triples pointed to interpretive content inside the gold answers themselves.

Interesting ideas

• I1. Comparing answers at the level of individual triples pinpoints exactly which relation is unsupported, instead of grading the whole answer as one unit.
• I2. The "gold-only" label marks reference content that the source text itself cannot back, which exposes interpretive or flawed gold annotations rather than blaming the system.
• I3. An "off-focus" label separates true statements that ignore what the question asked from statements that are actually false.
• I4. Humans adjudicate only the flagged, uncertain items, here about a quarter of all triples, which keeps manual effort small while handling annotation gaps.
• I5. Colour-coding the answer graph's edges by label gives reviewers a fast visual overview of where errors cluster within one answer.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): Readers answering fable or school-test questions could check each frame-derived claim of an answer against the world of the text and report which specific fact lacks support.
• I2 (learn): Evaluation could set aside gold items that the world model of the passage cannot derive, treating them as suspect references rather than as parser or reader failures.
• I3 (answer): The answerer could use the question's frame to filter world facts, then report "correct but irrelevant" output as its own category in test diagnostics.
• I4 (learn): Silver data could send to the LLM teacher or a human only those items where witness parsers or world checks disagree, leaving the agreed bulk automatic.
• I5 (world): A debugging view of the text world could colour entities and relations by provenance (stated, inferred, unsupported), making each reasoning step easy to inspect.