Elamur Research⌕
☰
home / papers / 2017 / Graph-based Neural Multi-Document Summarization

Graph-based Neural Multi-Document Summarization

Summary

Neural summarizers had worked well for single documents, but in multi-document summarization (MDS) earlier neural systems scored each sentence of a cluster in isolation. This ignored how sentences from different documents relate to one another. Yasunaga et al. add explicit sentence relation graphs to a neural extractive pipeline. Each sentence is encoded with a GRU (GRU sent). A Graph Convolutional Network (GCN) then propagates these embeddings over a graph of the whole cluster. A second GRU (GRU doc) builds a cluster embedding, and a regression layer scores salience. Sentences are then picked greedily, skipping redundant ones, until the length limit is reached. The authors compare three graphs: a tf-idf cosine similarity graph, the Approximate Discourse Graph (ADG) from G-Flow, and their new Personalized Discourse Graph (PDG). The PDG reweights ADG edges using sentence-level surface features. On DUC 2004 the model outperforms classic graph-based extractors and a GRU baseline that uses no graph. It is also competitive with state-of-the-art MDS systems. The work shows that graph structure can make up for scarce MDS training data and helps RNNs capture discourse across documents.

Key points

  • Evaluated on the DUC 2004 MDS task. The model splits the work into salience estimation and sentence selection, so it needs no costly sequence-to-sequence decoder.
  • The cosine graph connects two sentences when their tf-idf bag-of-words similarity is above 0.2. The ADG counts discourse cues such as co-reference, discourse markers, deverbal nouns and entity or event continuation.
  • ADG edge weights grow in steps of 0.5, which makes them coarse and inflates node degree. The PDG multiplies each weight by a linear-regression personalization score built from features such as position, length and proper-noun count, then normalizes it.
  • Node degree and salience correlate positively in all three graphs, with strength ordered PDG > ADG > cosine. The PDG also gave the lowest training and validation cost.
  • Choosing sentences purely by PDG node degree gave weak summaries, so the graphs are useful mainly as input to the GCN. A case study on cluster d30011t shows that salient sentences have high degree and are linked across documents.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

84 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k17-1045

@inproceedings{yasunaga2017graph,
  title = {{Graph-based Neural Multi-Document Summarization}},
  author = {Yasunaga, Michihiro and Zhang, Rui and Meelu, Kshitijh and Pareek, Ayush and Srinivasan, Krishnan and Radev, Dragomir},
  booktitle = {Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)},
  year = {2017},
  doi = {10.18653/v1/k17-1045},
  url = {https://doi.org/10.18653/v1/k17-1045}
}

References

47 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Réka Albert and Albert-László Barabási. 2002. Statistical mechanics of complex networks. Reviews of modern physics 74(1):47. \#\# Conclusion In this paper, we presented a novel multi-document summarization system that exploits the representational power of neural networks and graph representations… find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-Dan Gillick and Benoit Favre. 2009. A scalable global model for summarization. In Proceedings of the Workshop on Integer Linear Programming for Natural Langauge Processing. Association for Computational Linguistics, pages 10–18. gio. 2015. Neural machi… find
  3. Regina Barzilay, Noemie Elhadad, and Kathleen R McKeown. 2001. Sentence ordering in multidocument summarization. In Proceedings of the first international conference on Human language technology research. find
  4. Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In NAACL. find
  5. Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2017. Improving multi-document summarization via text classification. In AAAI. find
  6. David K Hammond, Pierre Vandergheynst, and Rémi Gribonval. 2011. Wavelets on graphs via spectral graph theory. Applied and Computational Harmonic Analysis 30(2):129–150. find
  7. Ziqiang Cao, Wenjie Li, Sujian Li, Furu Wei, and Yanran Li. 2016. Attsum: Joint learning of focusing and summarization with neural attention. In COLING. find
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  9. Ziqiang Cao, Furu Wei, Li Dong, Sujian Li, and Ming Zhou. 2015. Ranking with recursive neural networks and its application to multi-document summarization. In AAAI. find
  10. Kai Hong, John M Conroy, Benoit Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In LREC. find
  11. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In ACL. find our card
  12. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In EACL. find
  13. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In EMNLP. find
  14. Mikael Kågebäck, Olof Mogren, Nina Tahmasebi, and Devdatt Dubhashi. 2014. Extractive summarization using continuous vector space models. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)@ EACL. Citeseer, pages 31–39. find
  15. Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In NAACL. find
  16. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. find
  17. NIPS 2014 Deep Learning and Representation Learning Workshop . find
  18. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  19. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR. find our card
  20. John M Conroy, Judith D Schlesinger, Jade Goldstein, and Dianne P Oleary. 2004. Left-brain/right-brain multi-document summarization. In Proceedings of the Document Understanding Conference (DUC 2004). find
  21. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In ACL. find
  22. Chen Li, Xian Qian, and Yang Liu. 2013. Using supervised bigram-based ilp for extractive summarization. In ACL. find
  23. John M Conroy, Judith D Schlesinger, Jeff Kubina, Peter A Rankel, and Dianne P O’Leary. 2011. find
  24. Classy 2011 at tac: Guided and multi-lingual summaries and evaluation metrics. TAC 11:1–8. find
  25. Sujian Li, You Ouyang, Wei Wang, and Bin Sun. 2007. Multi-document summarization using support vector regression. In Proceedings of DUC. Citeseer. find
  26. John M Conroy, Judith D Schlesinger, and Dianne P O’Leary. 2006. Topic-focused multi-document summarization using an approximate oracle score. In COLING/ACL. find
  27. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse 8(3):243–281. find our card
  28. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS. find our card
  29. Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In European Conference on Information Retrieval. find
  30. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research 22:457–479. find
  31. Qiaozhu Mei, Jian Guo, and Dragomir Radev. 2010. Divrank: the interplay of prestige and diversity in information networks. In SIGKDD. find
  32. Rada Mihalcea and Paul Tarau. 2005. A language independent algorithm for single and multiple document summarization. In IJCNLP. find
  33. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP. find
  34. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  35. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  36. Xiaojun Wan and Jianwu Yang. 2006. Improved affinity graph based multi-document summarization. In NAACL. find
  37. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. AAAI . find our card
  38. Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In NAACL. find
  39. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL. find
  40. Wenpeng Yin and Yulong Pei. 2015. Optimizing sentence modeling and selection for document summarization. In IJCAI. find
  41. Ani Nenkova, Lucy Vanderwende, and Kathleen McKeown. 2006. A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization. In SIGIR. find
  42. Karolina Owczarzak, John M Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization. Association for Computational Linguistics, pages 1–9. find
  43. Thiago Pardo, Lucas Antiqueira, Maria Nunes, Osvaldo Oliveira, and Luciano da Fontoura Costa. 2006. Modeling and evaluating summaries using complex networks. Computational Processing of the Portuguese Language pages 1–10. find
  44. Dragomir R Radev. 2000. A common theory of information fusion from multiple text sources step one: cross-document structure. In Proceedings of the 1st SIGdial workshop on Discourse and dialogue-Volume 10. Association for Computational Linguistics, pages 74–83. find
  45. Dragomir R Radev, Timothy Allison, Sasha Blair-Goldensohn, John Blitzer, Arda Celebi, Stanko Dimitrov, Elliott Drabek, Ali Hakim, Wai Lam, Danyu Liu, et al. 2004a. Mead-a platform for multidocument multilingual text summarization. In LREC. find our card
  46. Dragomir R Radev, Sasha Blair-Goldensohn, Zhu Zhang, and Revathi Sundara Raghavan. 2001. Newsinessence: A system for domain-independent, real-time news clustering and multi-document summarization. In Proceedings of the first international conference on Human language technology research. Association… find
  47. Dragomir R Radev, Hongyan Jing, Małgorzata Styś, and Daniel Tam. 2004b. Centroid-based summarization of multiple documents. Information Processing & Management 40(6):919–938. find

↑ back to top

This page in Markdown →