Elamur Research⌕
☰
home / papers / 2026 / ContextCheck: Sentence-Level Faithfulness Verification with Context-Aware Disambiguation

ContextCheck: Sentence-Level Faithfulness Verification with Context-Aware Disambiguation

Paper · 2026 · Yueqin Yin, Yaxi Li, Xin Liu et al.

Summary

LLM outputs often contain claims that the supplied source does not support. Sentence-level checkers usually judge each sentence on its own, so they fail when a sentence depends on earlier ones, for example through pronouns or entities mentioned before. The usual fix is to rewrite such sentences into standalone claims before checking them, but this adds cost and introduces errors. ContextCheck instead verifies each sentence against the grounding document while also reading all previous sentences of the response, so references are resolved during verification itself.

The authors built training data from three context-dependent sources (MediaSum and MeetingBank dialogue summaries, plus RAGTruth- and ClearCheck-derived data) alongside standard single-sentence sets. They also proposed a taxonomy of 14 hallucination types for fine-grained error analysis. A Llama-3.1-8B-Instruct model fine-tuned on this data beats the strongest baselines by more than 10 Macro F1 points on the context-dependent benchmarks and stays competitive on conventional ones. This makes it a practical, low-overhead tool for detecting hallucinations at the sentence level.

Key points

  • Data pipeline:
    • Six models (GPT-4o, Claude-Sonnet-3.5, Llama-3.1-8B/3.2-3B, Qwen2.5-7B, Mistral-7B) wrote topic-focused summaries of MediaSum and MeetingBank dialogues.
    • GPT-4.1 flagged the context-dependent sentences.
    • Labels (Attributable / Not Attributable / Contradicted) were assigned through multi-LLM consensus.
  • Main results:
    • New state of the art on the three context-dependent sets (Context_RAGTruth, Context_MediaSum, Context_MeetingBank).
    • On 14 single-sentence datasets, average Macro F1 is 73.5, compared with 72.8 for the best prior 8B-scale verifiers.
  • Ablation on context: Removing preceding sentences consistently hurts accuracy. For example, ContextCheck-Direct drops from 92.24 to 88.6 Macro F1 on Context_MeetingBank.
  • Efficiency: On Context_MediaSum (623 samples, one H100), ContextCheck-Direct was compared with a FastCoref + ClearCheck-8B pipeline.
    • ContextCheck-Direct: 91.4 Macro F1 in 317s.
    • FastCoref + ClearCheck-8B: 72.4 Macro F1 in 340s.
    • Both use N LLM calls, whereas full LLM-based rewriting would need 2N.
  • Limitation: Training uses only SFT, and the CoT variant brings limited gains over the Direct variant. The authors suggest RLHF or DPO as future work. Code is available at github.com/yinyueqin/ContextCheck-Verifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States119 works · 57,950 citations🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yueqin Yin, Yaxi Li, Xin Liu, Xun Wang, Kaiqiang Song, Simin Ma, Shujian Liu, Sathish Reddy Indurthi, Haoyun Deng, Pengcheng He, Mingyuan Zhou and Song Wang. 2026. ContextCheck: Sentence-Level Faithfulness Verification with Context-Aware Disambiguation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.658

@inproceedings{yin2026contextcheck,
  title = {{ContextCheck: Sentence-Level Faithfulness Verification with Context-Aware Disambiguation}},
  author = {Yueqin Yin, Yaxi Li, Xin Liu, Xun Wang, Kaiqiang Song, Simin Ma, Shujian Liu, Sathish Reddy Indurthi, Haoyun Deng, Pengcheng He, Mingyuan Zhou and Song Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.658},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.658}
}

References

50 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, and 81 others. 2024. Internlm2 technical report. Preprint, arXiv:2403.17297.
  2. Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. Fintextqa: A dataset for long-form financial question answering. ACL. find
  3. Manuel Cossio. 2025. A comprehensive taxonomy of hallucinations in large language models. arXiv preprint arXiv:2508.01781. — [arXiv:2508.01781](https://arxiv.org/abs/2508.01781) find
  4. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In ICML.
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  6. Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. 2024. Anah-v2: Scaling analytical hallucination annotation of large language models. NeurIPS, 37:60012–60039. find
  7. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. NeurIPS, 28.
  8. Qisheng Hu, Quanyu Long, and Wenya Wang. 2025. Decomposition dilemmas: Does claim decomposition boost or burden fact-checking performance? NAACL. find
  9. Yebowen Hu, Tim Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. 2023. Meetingbank: A benchmark dataset for meeting summarization. ACL. find
  10. Alon Jacovi, Moran Ambar, Eyal Ben-David, Uri Shaham, Amir Feder, Mor Geva, Dror Marcus, and Avi Caciularu. 2024a. Coverbench: A challenging benchmark for complex claim verification. arXiv preprint arXiv:2408.03325. — [arXiv:2408.03325](https://arxiv.org/abs/2408.03325) find
  11. Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, and Mor Geva. 2024b. A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains. ACL.
  12. Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, and 1 others. 2025. The facts grounding leaderboard: Benchmarking llms’ ability to ground responses to long-form input. arXiv preprint arXiv:2501.03200.
  13. Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. 2024. Anah: Analytical annotation of hallucinations in large language models. ACL. find
  14. Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. Hover: A dataset for many-hop fact extraction and claim verification. arXiv preprint arXiv:2011.03088.
  15. Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. 2025. Why language models hallucinate. arXiv preprint arXiv:2509.04664. — [arXiv:2509.04664](https://arxiv.org/abs/2509.04664) find
  16. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. Wice: Real-world entailment for claims in wikipedia. ACL.
  17. Mahnaz Koupaee, Jake W Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, and 1 others. 2025. Faithful, unfaithful or ambiguous? multi-agent debate with initial stance for summary evaluation. NAACL. find
  18. Ádám Kovács and Gábor Recski. 2025. Lettucedetect: A hallucination detection framework for rag applications. arXiv preprint arXiv:2502.17125. find
  19. Deren Lei, Yaxi Li, Siyao Li, Mengya Hu, Rui Xu, Ken Archer, Mingyu Wang, Emily Ching, and Alex Deng. 2025. Factcg: Enhancing fact checkers with graph-based multi-hop data. NAACL. find
  20. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.
  21. Hongjun Liu, Yilun Zhao, Arman Cohan, and Chen Zhao. 2025a. Sucea: Reasoning-intensive retrieval for adversarial fact-checking through claim decomposition and editing. arXiv preprint arXiv:2506.04583. — [arXiv:2506.04583](https://arxiv.org/abs/2506.04583) find
  22. Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. EMNLP.
  23. Xin Liu, Lechen Zhang, Sheza Munir, Yiyang Gu, and Lu Wang. 2025b. Verifact: Enhancing long-form factuality evaluation with refined fact extraction and reference facts. EMNLP.
  24. Yining Lu, Noah Ziems, Hy Dang, and Meng Jiang. 2025. Optimizing decomposition for optimal claim verification. ACL. find
  25. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. Expertqa: Expert-curated questions and attributed answers. NAACL.
  26. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. EMNLP.
  27. Kushan Mitra, Dan Zhang, Sajjadur Rahman, and Estevam Hruschka. 2025. Factlens: Benchmarking fine-grained fact verification. ACL. find
  28. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ACL.
  29. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. ACL.
  30. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. ACL.
  31. Shon Otmazgin, Arie Cattan, and Yoav Goldberg. 2023. LingMess: Linguistically informed multi expert scorers for coreference resolution. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2752– 2760, Dubrovnik, Croatia. Association fo…
  32. Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, and Ying Ding. 2025. Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models. Preprint, arXiv:2502.14302. find
  33. Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, and Julia Belikova. 2025. When models lie, we learn: Multilingual span-level hallucination detection with psiloqa. ACL. find
  34. Wooseok Seo, Seungju Han, Jaehun Jung, Benjamin Newman, Seungwon Lim, Seungbeen Lee, Ximing Lu, Yejin Choi, and Youngjae Yu. 2025. Verifying the verifiers: Unveiling pitfalls and potentials in fact verifiers. COLM. find
  35. Yixiao Song, Yekyung Kim, and Mohit Iyyer. 2024. Veriscore: Evaluating the factuality of verifiable claims in long-form text generation. EMNLP.
  36. Liyan Tang, Tanya Goyal, Alexander R Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryś-ciński, Justin F Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. ACL.
  37. Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. Minicheck: Efficient fact-checking of llms on grounding documents. EMNLP.
  38. Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. Mediasum: A large-scale media interview dataset for dialogue summarization. NAACL.
  39. Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, and 1 others. 2024b. Tofueval: Evaluating hallucinations of llms on topic-focused dialogue summarization. NAACL. find
  40. Ziyu Zhuang. 2025. Pass-fc: Progressive and adaptive search scheme for fact checking of comprehensive claims. arXiv preprint arXiv:2504.09866. — [arXiv:2504.09866](https://arxiv.org/abs/2504.09866) find
  41. Vectara. 2024. Hhem v2: A new and improved factual consistency scoring model. Vectara Blog. find
  42. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. EMNLP.
  43. Song Wang, Xun Wang, Jie Mei, Yujia Xie, Si-Qing Chen, and Wayne Xiong. 2025. Developing a reliable, fast, general-purpose hallucination detection and mitigation service. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hu… find
  44. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, and 1 others. 2024. Factcheck-bench: Fine-grained evaluation benchmark for automatic fact-checkers. EMNLP. find
  45. Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, and Benjamin Van Durme. 2024a. A closer look at claim decomposition. ACL. find
  46. Miriam Wanner, Benjamin Van Durme, and Mark Dredze. 2024b. Dndscore: Decontextualization and decomposition for factuality verification in long-form text generation. arXiv preprint arXiv:2412.13175. find
  47. Stefan Webb and Michael Schuler. 2025. Introducing halloumi: A state-of-the-art claim-verification model. Oumi’s Blog. find
  48. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, and 1 others. 2024. Long-form factuality in large language models. Advances in Neural Information Processing Systems, 37:NeurIPS.
  49. Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and 1 others. 2025. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. arXiv preprint arXiv:2505.00212. — [arXiv:2505.00212](https://arxiv.org/abs/2505.00212) find
  50. Zhi Zheng and Wee Sun Lee. 2025. Reasoning-cv: Fine-tuning powerful reasoning llms for knowledge-assisted claim verification. arXiv preprint arXiv:2505.12348. — [arXiv:2505.12348](https://arxiv.org/abs/2505.12348) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yin-2026-contextcheck-sentence-level-faithfulness-verification/