Elamur Research
☰
home / papers / 2020 / FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization

FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.454.pdf

Summary

Neural abstractive summarizers often produce statements that the source article does not support, and standard metrics rarely catch these errors. Durmus, He and Diab study how faithful summaries are and propose FEQA, an automatic way to check it. They first had humans judge faithfulness for outputs from several systems (PGC, FastRL, BottomUp, TConv, BertSum) on CNN/DM and XSum. They also built a sentence-level scale of abstractiveness, running from copying a whole sentence to fusing pieces of several sentences. The judgments show a trade-off: the less a summary copies from the source, the more likely it is to contain errors. FEQA turns this into a test. It masks answer spans in each summary sentence and generates questions about them. A reading-comprehension model then answers those questions from the source document. If its answers don't match the masked spans, the summary probably contains unsupported content. FEQA was compared with metrics based on n-gram overlap, embeddings, entailment and relation extraction. It tracked human judgments best, and it was the only one that still did so on the highly abstractive XSum summaries. The result supports judging faithfulness separately from content selection.

Key points

  • Datasets: CNN/DM has about 288k training examples and XSum about 204k. Reference summaries in CNN/DM mostly compress source sentences by dropping words. No XSum reference sentence is copied from a single source sentence, and about 96% of its trigrams are new.
  • What drives abstractiveness: the training data matters more than the model architecture. The same systems are nearly extractive on CNN/DM but much more abstractive on XSum, though they still copy more than the human references.
  • How FEQA works: a learned model generates questions from masked spans of the summary. An off-the-shelf QA model answers them from the source, and the answers are compared using token F1. The authors say other QA models and answer types could be used in place of these.
  • Error analysis on 100 QA pairs: about 94% of the generated questions were acceptable. The QA model behaved correctly about 78% of the time. Most errors came from handling unanswerable questions wrongly, from long articles, and from exact-match F1 marking correct paraphrased answers as wrong.
  • ROUGE: ROUGE against the reference summary correlates only weakly with faithfulness on CNN/DM and not significantly on XSum. The authors still advise checking final results with human evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

245 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 227 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.454 · arXiv 2005.03754

@inproceedings{durmus2020feqa,
  title = {{FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization}},
  author = {Durmus, Esin and He, He and Diab, Mona},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.454},
  eprint = {2005.03754},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.454}
}

References

53 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Alec, W. Jeff, C. Rewon, L. David, A. Dario, and S. Ilya. 2019. Language models are unsupervised multitask learners. find
  2. Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lang… find our card
  3. Z. Cao, F. Wei, W. Li, and S. Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Association for the Advancement of Artificial Intelligence (AAAI). find
  4. A. Chaganty, S. Mussmann, and P. Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In Association for Computational Linguistics (ACL). find
  5. Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University. find
  6. P. Chen, F. Wu, T. Wang, and W. Ding. 2018. A semantic QA-based approach for text summarization evaluation. In Association for the Advancement of Artificial Intelligence (AAAI). find
  7. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Comp… find
  8. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In ACL. find our card
  9. J. Clarke and M. Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36. find
  10. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv, abs/1809.02922. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  12. M. Eyal, T. Baumel, and M. Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL). find
  13. T. Falke, L. F. R. Ribeiro, P. A. Utama, I. Dagan, and I. Gurevych. 2019a. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Association for Computational Linguistics (ACL). find our card
  14. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019b. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lingu… find our card
  15. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 363–370, Stroudsburg, PA, USA. Associati… find
  16. Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational AI. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 2–7, Melbourne, Australia. Association for Computational Linguistics. find
  17. M. Gardner, J. Berant, H. Hajishirzi, A. Talmor, and S. Min. 2019. Question answering is a format; when is it useful? arXiv preprint arXiv:1909.11291. — [arXiv:1909.11291](https://arxiv.org/abs/1909.11291) find
  18. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1– 6, Melbourne,… find our card
  19. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics. find our card
  20. B. Goodrich, V. Rao, P. J. Liu, and M. Saleh. 2019. Assessing the factual accuracy of generated text. In International Conference on Knowledge Discovery and Data Mining (KDD). find
  21. M. Grusky, M. Naaman, , and Y. Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In North American Association for Computational Linguistics (NAACL). find our card
  22. T. Hashimoto, H. Zhang, and P. Liang. 2019. Unifying human and statistical evaluation for natural language generation. In North American Association for Computational Linguistics (NAACL). find
  23. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, pages 1693–1701… find our card
  24. A. Holtzman, J. Buys, M. Forbes, and Y. Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. find
  25. R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard-Gontier, Y. Bengio, and J. Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In Association for Computational Linguistics (ACL). find our card
  26. I. Mani, G. Klein, L. Hirschman, T. Firmin, D. House, and B. Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In European Association for Computational Linguistics (EACL). find
  27. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics. find our card
  28. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Ma… find our card
  29. K. Knight and D. Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artifical Intelligence, 139:91– 107. find
  30. R. T. McCoy, E. Pavlick, and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007. find our card
  31. W. Kryściński, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. 2019. Neural text summarization: A critical evaluation. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  32. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI. find our card
  33. L. Lebanoff, J. Muchovej, F. Dernoncourt, D. S. Kim, S. Kim, W. Chang, and F. Liu. 2019. Analyzing sentence fusion in abstractive summarization. arXiv preprint arXiv:1910.00203. find
  34. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016b. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Associati… find
  35. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461. find our card
  36. J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110–119. find our card
  37. M. Li, S. Roller, I. Kulikov, S. Welleck, Y. Boureau, K. Cho, and J. Weston. 2019. Don’t say that! making inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:1911.03860. find
  38. C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Empirical Methods in Natural Language Processing (EMNLP). find
  39. Y. Liu and M. Lapata. 2019. Text summarization with pretrained encoders. In Empirical Methods in Natural Language Processing (EMNLP). find
  40. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find
  41. J. Novikova, O. Dušek, A. C. Curry, and V. Rieser. 2017. Why we need new evaluation metrics for NLG. In Empirical Methods in Natural Language Processing (EMNLP). find
  42. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Lin… find our card
  43. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
  44. T. Scialom, S. Lamprier, B. Piwowarski, and J. Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Empirical Methods in Natural Language Processing (EMNLP). find
  45. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  46. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In North American Association for Computational Linguistics (NAACL). find our card
  47. Z. Tu, Y. Liu, L. Shang, X. Liu, and H. Li. 2017. Neural machine translation with reconstruction. In Association for the Advancement of Artificial Intelligence (AAAI). find
  48. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76– 85, Berlin, Germany. Association for Computational Ling… find our card
  49. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692–2700. Curran Associates, Inc. find
  50. S. Wiseman, S. M. Shieber, and A. M. Rush. 2017. Challenges in data-to-document generation. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  51. F. Zhang, J. Yao, and R. Yan1. 2018. On the abstractiveness of neural document summarization. In Empirical Methods in Natural Language Processing (EMNLP). find
  52. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. 2019a. BERTSCORE: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. find
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. find