Elamur Research⌕
☰
home / papers / 2020 / Evaluating the Factual Consistency of Abstractive Text Summarization

Evaluating the Factual Consistency of Abstractive Text Summarization

Paper · 2020 · Wojciech Kryscinski, Bryan McCann, Caiming Xiong et al.

Summary

Abstractive summarization models often add claims to summaries that are absent from the source document or that contradict it. According to previous studies, this problem occurs in roughly a third of generated summaries. Popular metrics fail to detect it. The authors propose FactCC, a BERT-based model. It checks each summary sentence for consistency with the entire document rather than with individual source sentences. No labeled data existed for this task, so the training set was created artificially, that is, with weak supervision. Individual sentences were taken from news articles and modified according to rules. Some modifications preserve the meaning, for example back-translation. Others violate it: entity, number, and pronoun swapping, or negation. The extended version, FactCCX, additionally highlights the source span that supports the decision and the erroneous span in the sentence itself. On summaries produced by neural models for CNN/DailyMail, this approach outperforms models trained on NLI and fact checking. The highlighted spans also noticeably help human annotators. The work is important because it provides a scalable automatic method for verifying the factual accuracy of generated text.

Key points

  • The model is trained jointly on three tasks: determining whether a summary sentence is consistent with the source; finding the span in the document that supports this; and, for inconsistent sentences, pointing out the incorrect span.
  • For paraphrasing, back-translation through five languages was used: French, German, Chinese, Spanish, and Russian. The meaning-altering modifications are based on an analysis of typical summarizer errors: incorrect entity names, numbers, pronouns, and negation.
  • With the model-highlighted spans, annotators worked 21% faster (≈178 s vs. ≈225 s per task). Inter-annotator agreement measured by Fleiss' κ increased by approximately 38% (from 0.157 to 0.253). Over 79% of annotators found the highlights in the article useful.
  • The model-highlighted spans and independent human annotations matched with accuracy ≈65% and F1 0.62 for the article and 0.67 for the summary sentence.
  • Limitations: the model performs poorly at detecting commonsense errors, temporal and coreference inconsistencies, as well as highly abstractive summaries such as those in XSum. The authors released the data generation code, the model weights, and the manually annotated test set: github.com/salesforce/factCC.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SalesforceSan Francisco · United States56 works · 17,867 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSummarization · 232 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text summarization · 49 works

Cite

Cite as Wojciech Kryscinski, Bryan McCann, Caiming Xiong and Richard Socher. 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.750

@inproceedings{kryscinski2020evaluating,
  title = {{Evaluating the Factual Consistency of Abstractive Text Summarization}},
  author = {Wojciech Kryscinski, Bryan McCann, Caiming Xiong and Richard Socher},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.750},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.750}
}

References

44 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
  3. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th… find
  4. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In ACL (1), pages 675–686. Association for Computational Linguistics.
  5. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan…
  6. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational…
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  8. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. find
  9. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In HLT-NAACL.
  10. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. CoRR, abs/1808.09381. find
  11. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Conference of the Association for Computational Linguistic…
  12. Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-up abstractive summarization. In EMNLP, pages 4098–4109. Association for Computational Linguistics.
  13. Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019., pages 166–175. find
  14. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orlean…
  15. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Pape… find
  16. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. http://spacy.io . ↗
  17. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
  18. Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July…
  19. Ramesh Nallapati, Bowen Zhou, Çağlar Gülçehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. Proceedings of SIGNLL Conference on Computational Natural Language Learning.
  20. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NA…
  21. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
  22. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. CoRR, abs/1804.06451.
  23. Yichen Jiang and Mohit Bansal. 2018. Closed-book training to improve summarization encoder memory. In EMNLP, pages 4067–4077. Association for Computational Linguistics. find
  24. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. find
  25. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. CoRR, abs/1908.08960.
  26. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. In ICLR.
  27. Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In EMNLP, pages 1808–1817. Association for Computational Linguistics. find
  28. Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018. Improving neural abstractive document summarization with structural regularization. In EMNLP, pages 4078–4087. Association for Computational Linguistics. find
  29. Yang Liu. 2019. Fine-tune BERT for extractive summarization. CoRR, abs/1903.10318. find
  30. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. CoRR, abs/1908.08345.
  31. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. Proceedings of EMNLP.
  32. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. CoRR, abs/1511.06709. find
  34. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. CoRR, abs/1803.05355.
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Joshua Maynez, Shashi Narayan, Bernd Kaiser, and Illia Polosukhin. 2017. Attention is all Bohnet, and Ryan T. McDonald. 2020. you need. In Advances in Neural Information Pro-On faithfulness and factuality… find
  36. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS, pages 6297– 6308.
  37. Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. CoRR, abs/1809.01478. — [arXiv:1809.01478](https://arxiv.org/abs/1809.01478) find
  38. Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJC…
  39. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Transformers: State-of-the-art natural language processing.
  41. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional BERT contextual augmentation. In Computational Science -ICCS 2019 - 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part IV, pages 84–95. find
  42. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educa… find
  43. Fangfang Zhang, Jin-ge Yao, and Rui Yan. 2018. On the abstractiveness of neural document summarization. In EMNLP, pages 785–790. Association for Computational Linguistics. find
  44. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kryscinski-2020-evaluating-factual-consistency-abstractive-text/