Elamur Research⌕
☰
home / papers / 2026 / Revisiting Faithfulness Annotations for Long-form Summaries

Revisiting Faithfulness Annotations for Long-form Summaries

Paper · 2026 · Yang Zhong, Yang Janet Liu and Diane Litman

Summary

Benchmarks of long-form LLM summaries (four or more sentences) are widely used as gold references for faithfulness checkers. However, their annotation protocols mostly ask whether individual facts appear in the source. They rarely ask whether the links between facts are supported. The authors re-examine three such benchmarks: STORYSUMM (narratives), VERIGRAY (news) and FAREBIO (biomedical texts). These datasets differ in domain, annotator type, label taxonomy and aggregation scheme.

The audit has three stages:

  • Automated flagging: the RST-based framework DESCRIBE breaks summaries into atomic claims together with their discourse relations. Any sentence where its verdict conflicts with the gold label is flagged.
  • Complementary LLM analyses: additional LLM-based methods are applied to the flagged material.
  • Human verification: flagged cases are checked manually.

Between 3.4% and 5.4% of sentence labels turned out to need correction, mostly because of discourse-level problems. The authors group these into five error types. Fixing the labels changes how competing evaluators rank against each other. This suggests that noisy annotations can hide real performance differences, which matters for anyone building or comparing faithfulness metrics. The paper ends with four annotation recommendations and releases the corrected data.

Key points

  • Datasets audited:
    • STORYSUMM: 96 summaries, 580 sentences.
    • VERIGRAY: 412 summaries, 2,044 sentences, 7-category taxonomy.
    • FAREBIO: 175 summaries, 1,445 sentences. It is mostly singly annotated; only 34 sentences are doubly annotated, with Cohen's κ = 0.48.
  • Error taxonomy: the five categories are invented relational connections, mistakes of scope or attachment, mishandled timing or changes of state, dropped hedges, and semantic reframing. Narratives tend toward timing-related and relational mistakes, while news and science tend toward reframing and the loss of hedges.
  • STORYSUMM findings: 21 of 28 confirmed fixes were faithful labels that should have been unfaithful. These were often minority votes overridden by the unanimity rule. 15 of the 28 had been raised by at least one crowd worker.
  • Recommendations: use discourse-aware QA tools, review dissenting votes instead of relying on unanimity, pre-decompose claims, and add relation-level categories.
  • Resources and uptake: revised labels and code are released at github.com/cs329yangzhong/faithfulEval4LongSumm. The VERIGRAY authors have already updated their dataset based on this feedback.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PittsburghPittsburgh · United States65 works · 23,401 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yang Zhong, Yang Janet Liu and Diane Litman. 2026. Revisiting Faithfulness Annotations for Long-form Summaries. Proceedings of the 20th Linguistic Annotation Workshop (LAW XX).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.law-main.12

@inproceedings{zhong2026revisiting,
  title = {{Revisiting Faithfulness Annotations for Long-form Summaries}},
  author = {Yang Zhong, Yang Janet Liu and Diane Litman},
  booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)},
  year = {2026},
  doi = {10.18653/v1/2026.law-main.12},
  url = {https://doi.org/10.18653/v1/2026.law-main.12}
}

References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The datasets used in this paper are all publicly available for research purposes. We acknowledge the potential for bias in human evaluation. Responsible Disclosure. We contacted the authors of all three datasets prior to publication. The authors of S TORY S UMM (Subbiah et al., 2024) confirmed sever… — [arXiv:2510.21118](https://arxiv.org/abs/2510.21118) find
  2. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  3. Biaoyan Fang, Xiang Dai, and Sarvnaz Karimi. 2024. Understanding faithfulness and reasoning of large language models on plain biomedical summaries. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9890–9911, Miami, Florida, USA. Association for Computational Linguistic… find
  4. Gérard Genette. 1983. Narrative discourse : an essay in method / Gérardrard Genette ; translated by Jane E. Lewin ; foreword by Jonathan Culler. Cornell paperbacks. Cornell University Press, Ithaca, N.Y. find
  5. Ameya Godbole and Robin Jia. 2025. Verify with caution: The pitfalls of relying on imperfect factuality metrics. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22889–22912, Vienna, Austria. Association for Computational Linguistics. find
  6. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. SNaC: Coherence error detection for narrative summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 444–463, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  7. Sil Hamilton, Matthew Wilkens, and Andrew Piper. 2025. Narrabench: A comprehensive framework for narrative benchmarking. Preprint, arXiv:2510.09869. find
  8. Huan Yee Koh, Jiaxin Ju, He Zhang, Ming Liu, and Shirui Pan. 2022. How far are we from robust long abstractive summarization? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2682–2698, Abu Dhabi, United Arab Emirates. Association for Computational Lin… find
  9. Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguisti…
  10. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
  11. Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… find
  12. Yuho Lee, Taewon Yun, Jason Cai, Hang Su, and Hwanjun Song. 2024. UniSumEval: Towards unified, fine-grained, multi-dimensional summarization evaluation for LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3941– 3960, Miami, Florida, USA. Association for Computati… find
  13. William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243–281.
  14. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
  15. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  16. Wooseok Seo, Seungju Han, Jaehun Jung, Benjamin Newman, Seungwon Lim, Seungbeen Lee, Ximing Lu, Yejin Choi, and Youngjae Yu. 2025. Verifying the verifiers: Unveiling pitfalls and potentials in fact verifiers. In Second Conference on Language Modeling. find
  17. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. FineSurE: Fine-grained summarization evaluation using LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922, Bangkok, Thailand. Association for…
  18. Melanie Subbiah, Faisal Ladhak, Akankshya Mishra, Griffin Thomas Adams, Lydia Chilton, and Kathleen McKeown. 2024. STORYSUMM: Evaluating faithfulness in story summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9988–10005, Miami, Florida, U… find
  19. Melanie Subbiah, Akankshya Mishra, Grace Kim, Liyan Tang, Greg Durrett, and Kathleen McKeown. 2025. Is the top still spinning? evaluating subjectivity in narrative understanding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 185–203, Suzhou, China.… find
  20. Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association fo…
  21. Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label variation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269, Bangkok, Thai…
  22. Kun Zhang, Oana Balalau, and Ioana Manolescu. 2025. Structured discourse representation for factual consistency verification. In Findings of the Association for Computational Linguistics: ACL 2025, pages 820– 838, Vienna, Austria. Association for Computational Linguistics. find
  23. Yang Zhong and Diane Litman. 2025a. Discourse-driven evaluation: Unveiling factual inconsistency in long document summarization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long…
  24. Yang Zhong and Diane Litman. 2025b. A tale of evaluating factual consistency: Case study on long document summarization evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12511–12532, Vienna, Austria. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2026-revisiting-faithfulness-annotations-long-form/