Elamur Research⌕
☰
home / papers / 2019 / Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

Summary

Reinforcement learning lets abstractive summarizers optimise non-differentiable rewards, but the usual reward, ROUGE, measures n-gram overlap with a reference. It therefore favours lexical similarity and captures fluency poorly. The authors build on APES, which masks named entities in a reference summary to create cloze-style questions and then checks whether a QA model can answer them from the generated summary. They score summaries by QA F1 and by QA confidence (the probability given to the correct answer). They also add unsupervised variants that generate the questions from the source article, so no reference summary is needed.

Against human ratings of CNN/Daily Mail outputs, the unsupervised QA metrics track readability and relevance better than ROUGE, TextRank, Novelty or BERT LM perplexity. When used as RL rewards on top of the Gehrmann et al. (2018) model, they improve human-judged relevance without the readability collapse caused by a pure ROUGE reward. Because the metrics need no references, unlabeled in-domain and out-of-domain text can also be used for training.

Key points

  • The metrics are QA f score and QA conf, each in a supervised (sup) and an unsupervised (unsup) version, plus a learned combination ("QA learned") fitted by Ridge regression on 50 human-rated samples from Paulus et al. (2017).
  • Correlation study: the authors used 100 CNN/Daily Mail samples from 3 systems, rated 1–10 for Readability and Relevance on Amazon Mechanical Turk.
    • QA conf (unsup) reached Spearman ρ of 0.33 for readability and 0.31 for relevance.
    • ROUGE variants stayed at roughly 0.12–0.18.
  • Human evaluation against the baseline (readability 7.07, relevance 5.82):
    • A ROUGE-only reward dropped readability to 2.14.
    • QA learned trained with unlabeled CNN-DM TEST documents scored 6.89 for readability and 6.80 for relevance.
  • Effect of unlabeled data: benefits grew as the unlabeled text got closer to the test documents. TL;DR (out-of-domain) gave a slight gain, CNN-DM VAL a larger one, and CNN-DM TEST the largest.
  • Code: the metrics are to be released as the Python package summa-qa (github.com/recitalAI/summa-qa).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1320

@inproceedings{scialom2019answers,
  title = {{Answers Unite! Unsupervised Metrics for Reinforced Summarization Models}},
  author = {Scialom, Thomas and Lamprier, Sylvain and Piwowarski, Benjamin and Staiano, Jacopo},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1320},
  url = {https://doi.org/10.18653/v1/d19-1320}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kristjan Arumae and Fei Liu. 2019. Guiding extractive summarization with question-answering rewards. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2566–2577, M… find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  3. Ping Chen, Fei Wu, Tong Wang, and Wei Ding. 2018. A semantic qa-based approach for text summarization evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  4. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686, Melbourne, Australia. Association for Computational Ling… find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for C… find
  7. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. find our card
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693– 1701. find our card
  9. Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1808–1817, Brussels, Belgium. Association for Computational Linguistics. find
  10. Piji Li, Lidong Bing, and Wai Lam. 2018. Actor-critic based training framework for abstractive summarization. arXiv preprint arXiv:1803.11070. — [arXiv:1803.11070](https://arxiv.org/abs/1803.11070) find
  11. C-Y Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. of Workshop on Text Summarization Branches Out, Post Conference Workshop of ACL 2004. find
  12. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural L… find
  13. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. find
  14. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290. find
  15. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… find our card
  16. Ani Nenkova. 2011. Automatic Summarization. Foundations and Trends in Information Retrieval. Now. find
  17. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930. find
  18. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab. find our card
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  20. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646– 653, N… find our card
  21. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. find our card
  22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
  23. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732. find our card
  24. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024. find our card
  25. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. find
  26. Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, and Fang Chen. 2018. Summarization evaluation in the absence of human model summaries using the compositionality of word embeddings. In Proceedings of the 27th International Conference on Computational Linguistics, pages 905– 914, Santa Fe, New… find
  27. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  28. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. find
  29. Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl;dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59– 63. find our card
  30. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. find
  31. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270– 280. find
  32. Harris Wu, Dragomir R Radev, and Weiguo Fan. 2002. Towards answer-focused summarization. In Proceedings of the 1st International Conference on Information Technology and Applications. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/