Elamur Research⌕
☰
home / papers / 2020 / Factual Error Correction for Abstractive Summarization Models

Factual Error Correction for Abstractive Summarization Models

Summary

Abstractive summarizers built on pre-trained transformers often produce summaries that contradict their source documents. Earlier estimates put this at roughly 30% of outputs. The authors propose a post-hoc corrector rather than a new summarizer or another detection metric. A fine-tuned BART model receives a draft summary concatenated with its source article and generates a revised summary. Training data comes from CNN/DailyMail: reference summaries are corrupted with heuristic swaps of entities, numbers, dates and pronouns, chosen after manual inspection of real system errors. The same model also works as a consistency checker, since any edit it makes counts as an "inconsistent" verdict.

On artificial test data, the corrector restores most corrupted summaries exactly. On the human-annotated K2019 test set, it detects inconsistencies clearly better than FactCC and NLI-based baselines. However, it repairs only a small share of genuine system errors. This shows that synthetic corruptions do not reflect the variety of real summarizer mistakes, which is an important caveat for weakly supervised factuality work.

Key points

  • Training data: 30% (85,583) of the 287,227 CNN/DailyMail training summaries were corrupted, matching the estimated real error rate. The corruptions break down into 35,113 entity, 20,204 pronoun, 16,858 date and 13,408 number swaps. Swapped-in entities come from the source document, using spaCy NER.
  • Artificial test set: 62.13% of the 5,780 corrupted summaries were corrected to exactly match the reference. 73.73% of the 5,710 clean summaries were left unchanged. Consistency-checking accuracy was 84.38%.
  • K2019 consistency checking: 83.10% accuracy and 0.6121 macro F1, versus 74.15% and 0.5106 for FactCC. BERT+MNLI and BERT+FEVER scored only about 52% accuracy.
  • Human evaluation on real outputs:
    • Of 62 inconsistent summaries, the corrector edited 19 and fixed 11, a 17.74% success rate.
    • Of 441 consistent summaries, it broke the meaning of only 5 (1.13%).
  • Availability: code and data are released at github.com/mcao610/Factual-Error-Correction. The model is built on fairseq BART.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.506

@inproceedings{cao2020factual,
  title = {{Factual Error Correction for Abstractive Summarization Models}},
  author = {Cao, Meng and Dong, Yue and Wu, Jiapeng and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.506},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.506}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  2. Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  3. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics. find our card
  4. Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Association for Computational Linguistics. find
  5. Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisti… find our card
  6. Tian Shi, Ping Wang, and Chandan K. Reddy. 2019. LeafNATS: An open-source toolkit and live demo system for neural abstractive text summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 66–71, Minnea… find
  7. Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166–175. find
  8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  9. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687–697, Melbourne, Australia. Associat… find
  10. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228. find
  11. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840. find
  12. Krtin Kumar and Jackie Chi Kit Cheung. 2019. Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find
  13. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777. find our card
  14. Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2019b. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. arXiv preprint arXiv:1911.02541. find our card
  15. Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020. Boosting factual correctness of abstractive summarization with knowledge graph. arXiv preprint arXiv:2003.08612. — [arXiv:2003.08612](https://arxiv.org/abs/2003.08612) find
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find our card
  17. Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1430–1441. find
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  19. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/