Elamur Research⌕
☰
home / papers / 2021 / GO FIGURE: A Meta Evaluation of Factuality in Summarization

GO FIGURE: A Meta Evaluation of Factuality in Summarization

Paper · 2021 · Saadia Gabriel, Asli Celikyilmaz, Rahul Jha et al.

Summary

Neural summarizers often produce fluent text that misstates or invents facts. Many new automatic metrics aim to detect such errors, but there has been no systematic way to check whether these metrics actually work. GO FIGURE is a meta-evaluation framework that tests factuality metrics against five requirements: boundedness, sensitivity, robustness, generality and human correlation.

The authors built diagnostic data for three domains: XSUM, CNN/DailyMail and the dialogue corpus SAMSUM. Reference summaries were corrupted with controlled numbers of injected errors, and T5-generated summaries were annotated by hand for fine-grained error types. They then benchmarked ten metrics, covering QA-based, cloze-based, ROUGE and BERTScore approaches.

QA metrics, ROUGE-2/3 and BERTScore generally followed changes in factuality. ROUGE-1 and ROUGE-L often behaved invalidly, and cloze metrics were inconsistent across domains. Most metrics handled entity errors better than other error types. Metrics that failed on synthetic data also failed on model outputs, so the cheap controlled tests are a useful filter. Human-annotated data remains necessary for the most rigorous check.

Key points

  • Five conditions define a valid metric: Boundedness, Sensitivity, Robustness, Generality and Human Correlation.
    • The upper bound is the reference summary; the lower bound is a random summary from the corpus.
    • Sensitivity is measured with Pearson's r between the number of injected errors (1–3) and the average metric score.
  • Diagnostic data covers 500 source/summary pairs per domain (XSUM, CNNDM, SAMSUM).
    • The injected error types follow those observed in generated summaries, which account for 96.37% of all errors.
    • A further 250 XSUM and 250 SAMSUM T5 summaries were annotated by hand.
  • Ten metrics were tested: BLANC-Help, BLANC-Tune, SummaQA-C, SummaQA-F1, FEQA, ROUGE-1/2/3/L and BERTScore.
  • ROUGE-3 was the most consistently factuality-aware ROUGE variant across domains, while ROUGE-L can miss meaning flips such as replacing "increased" with "decreased".
  • The way QA metrics generate questions matters: questions drawn from the source (SummaQA) proved more robust across domains than questions drawn from the summary (FEQA).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Summarization · 159 works

Cite

Cite as Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi and Jianfeng Gao. 2021. GO FIGURE: A Meta Evaluation of Factuality in Summarization. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.42

@inproceedings{gabriel2021go,
  title = {{GO FIGURE: A Meta Evaluation of Factuality in Summarization}},
  author = {Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi and Jianfeng Gao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.42},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.42}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Ageeva, M. Forcada, Francis M. Tyers, and Juan Antonio Pérez-Ortiz. 2015. Evaluating machine translation for assimilation via a gap-filling task. In EAMT. find
  2. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics. our card
  3. Ziqiang Cao, Furu Wei, W. Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In AAAI. find
  4. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. ArXiv, abs/2006.14799. find
  5. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Melbourne, Australia. Association for Co… find
  6. Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. In ACL. our card
  7. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070, Online. Association for Computational Lingu… our card
  8. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long an… our card
  9. A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics, 9:391–409. find
  10. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019a. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In ACL. our card
  11. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019b. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lingu… our card
  12. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. ArXiv, abs/1911.12237. find
  13. B. Goodrich, V. Rao, Mohammad Saleh, and Peter J. Liu. 2019. Assessing the factual accuracy of generated text. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. find
  14. Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal. Association for Computational Linguistics. our card
  15. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687–697, Melbourne, Australia. Associat… find
  16. T. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. ArXiv, abs/1904.02792. find
  17. Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5094– 5107, Online. Association for Computational Linguistics. find
  18. Wojciech Kryscinski, B. McCann, Caiming Xiong, and R. Socher. 2019. Evaluating the factual consistency of abstractive text summarization. ArXiv, abs/1910.12840. find
  19. J. Landis and G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33 1:159–74. find
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. our card
  21. Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. ACL. our card
  22. Luca Massarelli, F. Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, F. Silvestri, and S. Riedel. 2019. How decoding strategies affect the verifiability of generated text. ArXiv, abs/1911.03587. find
  23. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41. our card
  24. Ramesh Nallapati, Bowen Zhou, C. D. Santos, Çaglar Gülçehre, and B. Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL. find
  25. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find
  26. Jekaterina Novikova, Ondrej Dusek, A. Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In EMNLP. our card
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683. our card
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. ArXiv, abs/1606.05250. our card
  29. Peter A. Rankel, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013. A decade of automatic content evaluation of news summaries: Reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 131–136, So… find
  30. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. find
  31. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In EMNLP/IJCNLP. our card
  32. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  33. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. find
  34. Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. EMNLP. find
  35. Oleg V. Vasilyev, Vedant Dharnidharka, and J. Bohannon. 2020. Fill in the blanc: Human-free quality estimation of document summaries. ArXiv, abs/2002.09836. our card
  36. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. find
  37. Tianyi Zhang, V. Kishore, Felix Wu, K. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. find
  38. Z. Zhao, Shay B. Cohen, and B. Webber. 2020. Reducing quantity hallucinations in abstractive summarization. ArXiv, abs/2009.13312. our card
  39. Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020. Boosting factual correctness of abstractive summarization. ArXiv, abs/2003.08612. — [arXiv:2003.08612](https://arxiv.org/abs/2003.08612) find
  40. Dataset Train Dev Test Domain XSUM CNNDM SAMSUM 204,045 287,227 14,732 11,332 13,368 818 11,334 11,490 819 Short news Long news Dialogues *Table 6: Summarization domains for evaluation.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gabriel-2021-go-figure-meta-evaluation-factuality/