Elamur Research⌕
☰
home / papers / 2021 / Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary

Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary

Paper · 2021 · Daniel Deutsch, Tania Bedrax-Weiss and Dan Roth

Summary

Most automatic metrics for summary content, such as ROUGE, BERTScore and MoverScore, compare a candidate with a reference at the token level. As a result, they tend to reward topical similarity rather than information the two texts actually share. The authors propose QAEval, a reference-based metric that treats content evaluation as question answering. Answer phrases (noun phrases) are chosen from the reference, and a fine-tuned BART model writes a question for each one. A QA model then tries to answer these questions using only the candidate summary. The score is the fraction of questions answered correctly, judged by exact match or token F1. The metric is tested on TAC'08, TAC'09 and a CNN/DailyMail subset, and each stage of the pipeline is analyzed separately. When ranking systems, QAEval agrees with human judgments better than other automatic metrics and matches the Pyramid Method. When scoring single summaries, it sometimes trails ROUGE. A manually annotated set of QA pairs suggests that human-level answering and answer checking would bring it close to Pyramid. This makes QA-based evaluation a credible companion to lexical metrics.

Key points

  • QAEval comes in two variants, QAEval-EM and QAEval-F1, and scores are macro-averaged when several references exist. It generalizes the earlier APES metric (Eyal et al., 2019), which only handled named entities.
  • The evaluation covers 92 instances from TAC'08 and TAC'09 plus 100 CNN/DailyMail instances. APES had originally been tested on just 8 TAC'11 document sets.
  • The two main bottlenecks are the QA model and the step that verifies whether a predicted answer is correct. An upper-bound estimate built from roughly 2.9k hand-annotated QA pairs approaches the gold-standard Pyramid Method.
  • On TAC'08, ROUGE variants correlate with each other at ≥ .80. Their correlation with the QA metrics is much lower: about .47 for QAEval-EM, .62 for QAEval-F1 and .26 for APES. The authors therefore recommend reporting a text-overlap metric and a QA-based metric together.
  • The main limitation is that QAEval relies on predicate-argument style questions (SQuAD-type QA), so it cannot capture more abstract content. Code is available at github.com/CogComp/qaeval-experiments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓University of PennsylvaniaPhiladelphia · United States170 works · 60,739 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Question answering · 242 works

Cite

Cite as Daniel Deutsch, Tania Bedrax-Weiss and Dan Roth. 2021. Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00397

@article{deutsch2021question,
  title = {{Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary}},
  author = {Daniel Deutsch, Tania Bedrax-Weiss and Dan Roth},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00397},
  url = {https://doi.org/10.1162/tacl_a_00397}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist… our card
  2. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2020. MOCHA: A dataset for training and evaluating generative reading comprehension metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16–20, 2020, pages 652… find
  3. Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 Update Summarization Task. In Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17–19, 2008. NIST. find
  4. Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2020. SummEval: Re-evaluating summarization evaluation. CoRR, abs/2007.12626v3. find
  5. Hoa Trang Dang and Karolina Owczarzak. 2009. Overview of the TAC 2009 Summarization Track. In Proceedings of the Text Analysis Conference. find
  6. Yanjun Gao, Chen Sun, and Rebecca J. Passonneau. 2019. Automated Pyramid summarization evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3–4, 2019, pages 404–418. Association for Computational Linguistics. find
  7. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. CoRR, abs/1809.02922v2. find
  8. Michael J. Denkowski and Alon Lavie. 2014. Meteor Universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26–27, 2014, Baltimore, Maryland, USA, pages 376–380. The Association for Compute… our card
  9. George Giannakopoulos, Vangelis Karkaletsis, George A. Vouros, and Panagiotis Stamatopoulos. 2008. Summarization system evaluation revisited: N-gram graphs. ACM Transactions on Speech and Language Processing, 5(3):5:1–5:39. https://doi.org/10.1145/1410358.1410359 — [doi:10.1145/1410358.1410359](https://doi.org/10.1145/1410358.1410359) ↗
  10. Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A Statistical analysis of summarization evaluation metrics using resampling methods. CoRR, abs/2104.00054v1. our card
  11. Eduard H. Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto. 2006. Automated summarization evaluation with basic elements. In Proceedings of the Fifth International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy, May 22–28, 2006, pages 899–902. European Language Resource… find
  12. Daniel Deutsch and Dan Roth. 2020. Understanding the extent to which summarization evaluation metrics measure the information quality of summaries. CoRR, abs/2010.12495v1. — [arXiv:2010.12495](https://arxiv.org/abs/2010.12495) find
  13. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, pages 5055–5070. Associati… our card
  14. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language genera… our card
  15. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. our card
  16. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15–20, 2018, Volume 2: Short Papers, pages 784–789. Associa… our card
  17. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267–300. https://doi.org/10.1162/COLI a 00123 ↗ our card
  18. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, pages 2383–2392. The As… our card
  19. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August… find
  20. Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight Pyramids for manual summary evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis… find
  21. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… our card
  22. Stephen Tratz and Eduard H. Hovy. 2008. Summarization evaluation using transformed basic elements. In Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17–19, 2008. NIST. find
  23. Ani Nenkova and Rebecca J. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2–7, 2004, pages 145… find
  24. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, pages 5008–5020. Association for Computation… find
  25. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 74–84. Association for Computational… find
  26. Qian Yang, Rebecca J. Passonneau, and Gerard de Melo. 2016. PEAK: Pyramid Evaluation via Automated Knowledge Extraction. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12–17, 2016, Phoenix, Arizona, USA, pages 2673–2680. AAAI Press. find
  27. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net. find
  28. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deutsch-2021-question-answering-as-automatic-metric/