Elamur Research
☰
home / papers / 2019 / Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference

Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1213.pdf

Summary

Abstractive summarizers write fluent text, but they often state things the source article does not support. This paper measures how common such errors are in single-document summarization on CNN-DM and tests whether natural language inference (NLI) can catch them. The authors built a crowdsourcing protocol in which workers label each summary sentence against the article, with labels merged by MACE. Using it, they found that roughly a quarter of the outputs from the strongest systems (FAS, BUS) contain errors.

They then reranked beam-search alternatives by entailment. Each candidate is scored by taking every summary sentence's best entailment probability against any source sentence and averaging those values. Five NLI models were compared. Off-the-shelf models gave small or inconsistent gains. ESIM even increased errors on the test set, and only BERT helped substantially. A simpler test of choosing between correct and incorrect sentence pairs showed a large gap to human accuracy.

The main lesson is that strong benchmark NLI scores do not carry over to this task. The released annotations offer a new extrinsic test set for NLI.

Key points

  • Error rates on 100 CNN-DM test summaries (9 crowdworkers each): PGC 8%, FAS 26%, BUS 25%. Correctness did not track ROUGE, and the more abstractive systems made more errors.
  • Crowdsourcing reliability: agreement with an expert gold standard (50 summaries, 140 sentences) became substantial with 3 workers and plateaued at 9, with Cohen's κ of 0.74.
  • NLI models tested: DA and ESIM (AllenNLP, with ELMo, trained on SNLI), InferSent, SSE, and BERT-base fine-tuned on MultiNLI.
  • Reranking results: on 200 validation documents (5 FAS hypotheses each), the beam-order baseline had 42.1% incorrect summaries. BERT gave the clearest gain but still left over half of the errors. On the test set, ESIM reranking raised errors from 26% to 29%.
  • Sentence-pair test (373 triples): the share of pairs where the incorrect sentence was preferred was 50% for random choice, 32.4% for ESIM (best model), 35.9% for BERT, and 16.1% for humans. Likely causes are domain shift and reliance on lexical overlap. The data are released via TUdatalib.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

184 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 169 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1213

@inproceedings{falke2019ranking,
  title = {{Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference}},
  author = {Falke, Tobias and Ribeiro, Leonardo F. R. and Utama, Prasetya Ajie and Dagan, Ido and Gurevych, Iryna},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1213},
  url = {https://doi.org/10.18653/v1/p19-1213}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. find
  2. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the Original: Fact Aware Neural Abstractive Summarization. In Proceedings of the Thiry-Second AAAI Conference on Artificial Intelligence, pages 4784–4791, New Orleans, LA, USA. find
  3. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1657–1668, Vancouver, Canada. find
  4. Yen-Chun Chen and Mohit Bansal. 2018. Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 675–686, Melbourne, Australia. find our card
  5. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive Sentence Summarization with Attentive Recurrent Neural Networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93– 98, San D… find our card
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,… find our card
  7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelber… find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Lisa Fan, Dong Yu, and Lu Wang. 2018. Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information. In NIPS 2018 Interpretability and Robustness for Audio, Speech and Language Workshop, Montreal, Canada. find
  10. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-Up Abstractive Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. find our card
  11. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 687– 697, Melbourne, Australia. find
  12. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 1693–1701, Montreal, Canada. find our card
  13. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, GA, USA. find our card
  14. Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1430–1441, Santa Fe, NM, USA. find
  15. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A Simple, Fast Diverse Decoding Algorithm for Neural Generation. ArXiv preprint 1611.08562. — [arXiv:1611.08562](https://arxiv.org/abs/1611.08562) find
  16. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. find our card
  17. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. find
  18. Yixin Nie and Mohit Bansal. Shortcut-Stacked Sentence Encoders for Multi-Domain Inference. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 41–45, Copenhagen, Denmark. find
  19. Nangia Nikita and Samual R. Bowman. 2019. Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. find
  20. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255, Austin, Texas. find our card
  21. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-Reward Reinforced Summarization with Saliency and Entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 646–653, New Orleans, LA, USA. find
  22. Ramakanth Pasunuru, Han Guo, and Mohit Bansal. 2017. Towards Improving Abstractive Summarization via Entailment Generation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 27–32, Copenhagen, Denmark. find
  23. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  24. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. find
  25. Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, Reykjavik, Iceland. find
  26. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1073–1083, Vancouver, Canada. find
  27. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112–1122,… find