Elamur Research
☰
home / papers / 2019 / Hierarchical Transformers for Multi-Document Summarization

Hierarchical Transformers for Multi-Document Summarization

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1500.pdf

Summary

The paper addresses abstractive multi-document summarization. Neural models struggle here because training data is scarce and input clusters are too large to encode in full. Building on the WikiSum setup of Liu et al. (2018), the input is a Wikipedia title plus cited web pages and Google search results, and the target is the article's lead section.

The authors propose a two-stage system:

  • A learned ranker scores source paragraphs, and only the top-ranked ones are passed on.
  • A Hierarchical Transformer summarizes the selected paragraphs. Unlike earlier work, it does not flatten them into a single long sequence. It stacks local (within-paragraph) and global (across-paragraph) layers, so information is exchanged between paragraphs through attention.

The global attention can learn latent links between paragraphs. It can also be guided by explicit graphs based on lexical similarity or discourse relations.

On WikiSum, the model outperforms strong baselines, including a Flat Transformer and T-DMCA, in both automatic and human evaluation. The work shows that modelling document structure and learning which content to select both matter for scaling neural summarization to many sources.

Key points

  • Data: WikiSum, with hundreds of thousands of instances. Sources are pages cited in a Wikipedia article plus the top 10 Google results for its title, split into paragraphs at line breaks.
  • Ranker: two LSTM encoders (title and paragraph), max-pooling, and a sigmoid output. Each paragraph's training target is its ROUGE-2 recall against the gold lead section. This replaces the tf-idf cosine ranking used by Liu et al. (2018).
  • Encoder: freely stackable local and global Transformer layers. Inter-paragraph attention can be swapped for a graph matrix built from similarity or discourse relations. Decoding uses beam search with a length penalty.
  • Human evaluation: Amazon Mechanical Turk with 5 responses per HIT, using a QA-based test and best–worst ranking (scores from −1 to 1). HT was clearly preferred over Lead, Flat Transformer and T-DMCA, and all pairwise differences were significant (ANOVA with Tukey HSD, p < 0.01).
  • Resource: code and data are released at github.com/nlpyang/hiersumm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Discourse analysis

Cited by

140 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 139 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1500

@inproceedings{liu2019hierarchical,
  title = {{Hierarchical Transformers for Multi-Document Summarization}},
  author = {Liu, Yang and Lapata, Mirella},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1500},
  url = {https://doi.org/10.18653/v1/p19-1500}
}

References

41 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In In Proceedings of the 3rd International Conference on Learning Representations, San Diego, California. find our card
  3. Regina Barzilay and Kathleen R. McKeown. 2005. Sentence fusion for multidocument news summarization. Computational Linguistics, 31(3):297– 327. find
  4. Lidong Bing, Piji Li, Yi Liao, Wai Lam, Weiwei Guo, and Rebecca Passonneau. 2015. Abstractive multi-document summarization via phrase selection and merging. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natura… find
  5. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p… find
  6. Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163–1173, Atlanta, Georgia.… find
  7. Eric Chu and Peter J Liu. 2018. Unsupervised neural multi-document abstractive summarization. arXiv preprint arXiv:1810.05739. — [arXiv:1810.05739](https://arxiv.org/abs/1810.05739) find
  8. James Clarke and Mirella Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36(3):411–441. find
  9. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  10. Katja Filippova and Michael Strube. 2008. Sentence fusion via dependency graph compression. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 177– 185, Honolulu, Hawaii. find
  11. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. find our card
  12. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa… find our card
  13. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693– 1701. Curran Associates, Inc. find our card
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  15. Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France. find
  16. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  17. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico. find our card
  18. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 465– 470, Vancouver, Canada. find
  19. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479. find
  20. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. find our card
  21. Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Structured neural summarization. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, Louisiana. find
  22. Logan Lebanoff and Fei Liu. 2018. Automatic detection of vague words and sentences in privacy policies. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3508–3517, Brussels, Belgium. find
  23. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  24. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by summarizing long sequences. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada. find our card
  25. Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Association for Computational Linguistics, 6:63–75. find
  26. Jordan J Louviere, Terry N Flynn, and Anthony Alfred John Marley. 2015. Best-worst scaling: Theory, methods and applications. Cambridge University Press. find
  27. Shulei Ma, Zhi-Hong Deng, and Yunlun Yang. 2016. An unsupervised multi-document summarization framework based on neural document model. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1514–1523, Osaka, Japan. find
  28. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… find
  29. Vlad Niculae, André F. T. Martins, and Claire Cardie. 2018. Towards dynamic computation graphs via sparse latent structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 905–911, Brussels, Belgium. find
  30. Daraksha Parveen and Michael Strube. 2014. Multi-document summarization using bipartite graphs. In Proceedings of TextGraphs-9: the workshop on Graph-based Methods for Natural Language Processing, pages 15–24, Doha, Qatar. find
  31. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada. find our card
  32. Dragomir Radev. 2000. A common theory of information fusion from multiple text sources step one: Cross-document structure. In 1st SIGdial Workshop on Discourse and Dialogue, pages 74–83, Hong Kong, China. find
  33. Evan Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia, 6(12). find
  34. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. find
  35. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  37. Xiaojun Wan. 2008. An exploration of document impact on graph-based multi-document summarization. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 755–762, Honolulu, Hawaii. find
  38. Adina Williams, Andrew Drozdov, and Samuel R. Bowman. 2018. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association for Computational Linguistics, 6:253–267. find
  39. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. In arXiv preprint arXiv:1609.08144. find
  40. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452–462, Vancouver, Canada. find
  41. Jianmin Zhang, Jiwei Tan, and Xiaojun Wan. 2018. Adapting neural single-document summarization model for abstractive multi-document summarization: A pilot study. In Proceedings of the International Conference on Natural Language Generation. find