Elamur Research
☰
home / papers / 2018 / Controllable Abstractive Summarization

Controllable Abstractive Summarization

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-2706.pdf

Summary

Most neural summarizers produce a single summary per document and ignore what the reader wants. Fan, Grangier and Auli address this with a model whose output can be shaped by the user. They control length, focus on particular entities, the style of a specific news source (CNN vs. DailyMail), and summarizing only the part of an article not yet read. The base system is a convolutional sequence-to-sequence network (Gehring et al., 2017) with gated linear units, multi-hop encoder attention and decoder self-attention. Instead of a pointer/copy mechanism, it uses BPE subwords and shared embeddings. Each control is a special marker token prepended to the input; at training time the markers are taken from the reference summary. For remainder summarization, the authors create synthetic training data by aligning reference sentences to article sentences using ROUGE-L. When users give no input, the controls are set to values tuned on held-out data. Even then, the model beats pointer-based maximum-likelihood systems on CNN/DailyMail and is preferred by human raters. This shows that conditioning variables can improve summary quality as well as enable personalization.

Key points

  • Using the full-text CNN/DailyMail data, the model achieves an F1-ROUGE-1 of 40.38, whereas See et al. (2017) reach 39.53. On the entity-anonymized version, it gains +0.76 ROUGE-1.
  • Length is split into discrete bins, each holding roughly equal numbers of training documents. Entities are anonymized as @entity0…@entityN, and source styles are encoded as @genSource tokens.
  • For remainder summarization, a model given both the read portion and the remainder reaches 30.70 ROUGE-1 when length control is added, versus 28.12 for a standard full-summary baseline.
  • In a blind human study (500 test articles, 5 raters each, ~2.5k judgments), raters preferred the model's summaries about 59% of the time.
  • Experiments were run with the fairseq toolkit, and the length control outperforms the approach of Kikuchi et al. (2016).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

151 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 141 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-2706

@inproceedings{fan2018controllable,
  title = {{Controllable Abstractive Summarization}},
  author = {Fan, Angela and Grangier, David and Auli, Michael},
  booktitle = {Proceedings of the 2nd Workshop on Neural Machine Translation and Generation},
  year = {2018},
  doi = {10.18653/v1/w18-2706},
  url = {https://doi.org/10.18653/v1/w18-2706}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Catherine Kobus, Josep Crego, and Jean Senellart. 2017. Domain control for neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017. INCOMA Ltd., Varna, Bulgaria, pages 372–378. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representation (ICLR) . find our card
  3. Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Fader networks: Manipulating images by sliding attributes. Arxiv abs/1706.00409. — [arXiv:1706.00409](https://arxiv.org/abs/1706.00409) find
  4. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CoNLL. find our card
  5. Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) . find our card
  6. Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. 1990. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems (NIPS). pages 396–404. find
  7. Dipanjan Das and André FT Martins. 2007. A survey on automatic text summarization. Literature Survey for the Language and Statistics II course at CMU 4:192–195. find
  8. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out. find
  9. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks . find our card
  10. H. P. Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development 2(2). find
  11. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. Arxiv abs/1511.06114. find our card
  12. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. Workshop on Stylistic Variations abs/1707.02633. find our card
  13. Katja Filippova. 2017. Sentence and passage summarization for question answering. Talk given at EMNLP 2017 Workshop on New Frontiers in Summarization. find
  14. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015b. Effective approaches to attention-based neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  15. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 . — [arXiv:1508.06576](https://arxiv.org/abs/1508.06576) find
  16. Kathleen McKeown. 1992. Text generation. Cambridge University Press. find
  17. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning . find our card
  18. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. Conference of the Association for the Advancement of Artificial Intelligence (AAAI) . find our card
  19. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. find
  20. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. Conference of the Association for the Advancement of Artificial Intelligence (CONLL) . find
  21. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Neural Information Processing Systems (NIPS). find our card
  22. Ani Nenkova, Kathleen McKeown, et al. 2011. Automatic summarization. Foundations and Trends R in Information Retrieval 5(2–3):103–233. find
  23. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In ICML. find
  24. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In ICML. find our card
  25. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 . find our card
  26. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. arXiv preprint arXiv:1609.09552 . find
  27. Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Joseph Pal, and Aaron C. Courville. 2017. Adversarial generation of natural language. Workshop on Representation Learning for NLP . find
  28. Diederik P. Kingma and Max Welling. 2013. Autoencoding variational bayes. Arxiv abs/1312.6114. find our card
  29. Alexander M Rush, SEAS Harvard, Sumit Chopra, and Jason Weston. 2015. A neural attention model for sentence summarization. Conference on Empirical Methods in Natural Language Processing (EMNLP) . find
  30. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. Annual Meeting of the Association for Computational Linguistics (ACL) . find
  31. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Compu… find our card
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. Annual Meeting of the Association for Computational Linguistics (ACL) . find
  33. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. Arxiv abs/1705.09655. find our card
  34. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. 2013. On the importance of initialization and momentum in deep learning. In ICML. find
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Neural Information Processing Systems (NIPS). find
  36. Jun Suzuki and Masaaki Nagata. 2017. Cutting-off redundant repeating generations for neural abstractive summarization. In European Conference of the Association of Computer Linguists (EACL). find
  37. Shunsuke Takeno, Masaaki Nagata, and Kazuhide Yamamoto. 2017. Controlling target features in neural machine translation via prefix constraints. In Proceedings of the 4th Workshop on Asian Translation (WAT2017). Asian Federation of Natural Language Processing, Taipei, Taiwan, pages 55–63. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762 . find
  39. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015a. Pointer networks. In Conference on Advances in Neural Information Processing Systems (NIPS). find
  40. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015b. Show and tell: A neural image caption generator. Conference on Computer Vision and Pattern Recognition (CVPR) . find our card
  41. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI. find
  42. Junbo Zhao, Y. Kim, K. Zhang, A. M. Rush, and Y. Le-Cun. 2017. Adversarially Regularized Autoencoders for Generating Discrete Structures. ArXiv 1706.04223. — [arXiv:1706.04223](https://arxiv.org/abs/1706.04223) find