📄 Localizing Factual Inconsistencies in Attributable Text Generation

Summary

LLM outputs that should be grounded in a reference document, such as summaries, open-book answers or cited responses, still contain hallucinations. Existing detectors usually flag whole sentences, spans, or "atomic" claims that bundle several facts together, so they cannot show exactly which piece of information is wrong. The authors propose QASemConsistency. Drawing on Neo-Davidsonian semantics, it splits a generated text into minimal propositions. Each proposition is one predicate–argument relation, written as a simple question–answer pair in the QA-SRL (verbs) and QANom (nominalizations) formats. Each pair is then judged as supported or unsupported by the reference. Because the pairs are plain natural language, crowd workers can label them. The authors built a benchmark of over 3K instances covering several attributable generation tasks, with substantial inter-annotator agreement. They also show that the resulting consistency scores track human judgments. Automatic detection was tested with supervised NLI models and prompted open and commercial LLMs. These handle such fine-grained hypotheses reasonably well but leave much room for improvement. The work offers a precise, human-usable unit for evaluating and correcting hallucinations.

Key points

• Text is decomposed via QA-SRL and QANom, so each QA pair encodes a single semantic relation. It is compared against token-level (CLIFF), QGQA (Q²), claim-level (FActScore), and structured approaches (DAE, FactGraph).
• In 27% of predicates, some of their QA pairs were supported and others were not. Coarser claim-level units would miss this distinction.
• The crowdsourced benchmark has more than 3K instances. For contrast, span highlighting in CLIFF reached only 0.35 Fleiss Kappa.
• Detectors were built from NLI models trained on standard entailment data and from open-source and commercial LLMs. Code, data and models are released on the first author's GitHub (github.com/ariecattan).
• Limitations: only verbal and nominal predicates are covered, so errors in adjectives or discourse relations are not isolated at the finest level. The authors suggest QADiscourse or QA-Adj as possible extensions. Published in TACL vol. 14 (2026).