Elamur Research
☰
home / papers / 2021 / Annotating and Modeling Fine-grained Factuality in Summarization

Annotating and Modeling Fine-grained Factuality in Summarization

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.naacl-main.114.pdf

Summary

Abstractive summarizers often produce statements the source document does not support. It has been unclear which kinds of errors matter most and whether current factuality detectors catch them. Goyal and Durrett compare three ways to get training data for factuality classifiers on two English news domains, XSum and CNN/DailyMail. These are entity-centric synthetic corruptions (Ent-C, following Kryscinski et al., 2020), generation-centric paraphrase data (Gen-C), and human-labeled model summaries. To compare them fairly, the authors align supervision at the sentence, dependency-arc and word levels, deriving arc labels from highlighted error spans. Error profiles differ sharply between the two domains. Rule-based corruptions also poorly match the mistakes real models make on the highly abstractive XSum data. Fine-grained human annotation gives the strongest training signal: it lets the dependency-arc entailment (DAE) model detect and locate errors better than sentence-level or synthetic supervision. As a practical use, the DAE model flags non-factual tokens in noisy reference summaries so they can be excluded from the training loss. The resulting BART summarizer is noticeably more faithful.

Key points

  • Synthetic data: about 50k Ent-C pairs per domain, built with entity, number and pronoun swaps, negation, noise injection and backtranslation. Gen-C provides 40k examples per domain, using 10th-best paraphrases from the Goyal and Durrett (2020b) model.
  • An arc counts as non-factual if its head or child word falls inside a highlighted error span. Word-level decisions can be recovered from arc predictions.
  • On XSum, the DAE model trained on fine-grained human labels localizes errors better than DAE-Weak at both the arc and word level.
  • The authors mask DAE-flagged tokens out of the log-likelihood loss when training BART (initialized from BART-XSUM-LARGE, 50k examples). Compared with a standard baseline and loss truncation, this cuts the word error rate from 32.2 (baseline) and 31.1 (loss truncation) to 23.7. The sentence error rate falls from 74.0 and 70.9 to 61.4.
  • In a Mechanical Turk evaluation, 7 annotators each judged summaries for 50 articles. Factuality scores were 46.5 for the masked model, versus 37.4 for the baseline and 39.1 for loss truncation. Code and data are released at github.com/tagoyal/factuality-datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

116 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.114

@inproceedings{goyal2021annotating,
  title = {{Annotating and Modeling Fine-grained Factuality in Summarization}},
  author = {Goyal, Tanya and Durrett, Greg},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.114},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.114}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7352–7364. find
  2. Regina Barzilay, Kathleen R. McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 550–557, College Park, Maryland, USA. Association for Computational Li… find
  3. Taylor Berg-Kirkpatrick, Dan Gillick, and Dan Klein. 2011. Jointly Learning to Extract and Compress. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 481–490. find
  4. Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251–6258. find
  5. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  6. Giuseppe Carenini and Jackie C. K. Cheung. 2008. Extractive vs. NLG-based abstractive summarization of evaluative text: The effect of corpus controversiality. In Proceedings of the Fifth International Natural Language Generation Conference, pages 33– 41, Salt Fork, Ohio, USA. Association for Computa… find
  7. Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2007. Guiding semi-supervision with constraint-driven learning. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 280–287. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1998–2008. find
  10. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics (TACL). find
  11. Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisti… find our card
  12. Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. 2010. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 11:2001–2049. find
  13. Dan Gillick and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 148–151, Los Angeles. Association for Computational Linguistics. find
  14. Tanya Goyal and Greg Durrett. 2020a. Evaluating factuality in generation with dependency-level entailment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3592–3603. find our card
  15. Tanya Goyal and Greg Durrett. 2020b. Neural syntactic preordering for controlled paraphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 238– 252, Online. Association for Computational Linguistics. find our card
  16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701. find our card
  17. Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320–9331. find
  18. Daniel Kang and Tatsunori Hashimoto. 2020. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718–731, Online. Association for Computational Linguistics. find
  19. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. In Proceedings of the 58th Annual Meeting Kevin Knight and Daniel Marcu. 2000. Statistics-Based Summarization—Step One: Sentence Compression. In Proceedi… find our card
  20. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346. find
  21. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6859–6866. find
  22. Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Analyzing sentence fusion in abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, Hong Kong, China. Association for Computational Linguistics. find
  23. Kaiqiang Song, Logan Lebanoff, Qipeng Guo, Xipeng Qiu, Xiangyang Xue, Chen Li, Dong Yu, and Fei Liu. 2020a. Joint parsing and generation for abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8894–8901. find
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  25. Kaiqiang Song, Bingqing Wang, Zhe Feng, Ren Liu, and Fei Liu. 2020b. Controlling the amount of verbatim copying in abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8902–8909. find
  26. Klaus-Michael Lux, Maya Sappelli, and Martha Larson. 2020. Truth or Error? Towards systematic analysis of factual errors in abstractive summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 1–10, Online. Association for Computational Linguistics. find
  27. Yuning Mao, Xiang Ren, Heng Ji, and Jiawei Han. 2020. Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation. In arXiv preprint 2010.12723. — [arXiv:2010.12723](https://arxiv.org/abs/2010.12723) find
  28. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan Thomas Mcdonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics (ACL). find
  29. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  30. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684. — [arXiv:1910.08684](https://arxiv.org/abs/1910.08684) find
  31. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find
  32. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290. find
  33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  34. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. find
  35. Kristian Woodsend and Mirella Lapata. 2012. Multiple Aspect Summarization Using Integer Linear Programming. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing (EMNLP) and Computational Natural Language Learning (CoNLL). find
  36. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Fang-Fang Zhang, Jin-ge Yao, and Rui Yan. 2018. On the abstractiv… find
  37. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the International Conference on Machine Learning. find our card
  38. Sent-Factuality DAE DAE-Weak 96.4 95.4 94.8 91.2 97.3 97.8 96.1 94.3 95.3 97.9 97.1 95.9 XS UM Sent-Factuality DAE DAE-Weak *Table 7: Performance of factuality models when trained on synthetic factuality training datasets on their held-out test sets.* Zheng Zhao, Shay B Cohen, and Bonnie Webber. 202… find
  39. Synthetic Dataset Performance on held-out samples Manual Annotation of Errors find