Elamur Research⌕
☰
home / papers / 2020 / Heterogeneous Graph Neural Networks for Extractive Document Summarization

Heterogeneous Graph Neural Networks for Extractive Document Summarization

Summary

Extractive summarization depends on modelling how sentences relate to each other. RNN encoders struggle with long-range links, and similarity or discourse graphs depend on heuristics or external parsers that can introduce errors. Fully connected sentence graphs, such as Transformer encoders, leave open which structure works best. The authors propose HeterSUMGraph, which represents a document as a graph with two node types, words and sentences. Sentences are never linked directly; they connect only through shared words, and TF-IDF values serve as edge features. A CNN plus BiLSTM encoder initializes the sentence nodes. A Graph Attention Network (GAT) with residual connections and feed-forward layers then alternately refreshes word and sentence representations. Finally, a classifier picks sentences, trained on greedy ROUGE-based ORACLE labels. Adding document nodes yields HeterDocSUMGraph for multi-document input. Without any pre-trained language model, the system reportedly outperforms earlier approaches on CNN/DailyMail, NYT50 and Multi-News. This offers a simple, extensible way to add intermediate semantic units to graph summarizers.

Key points

  • On the CNN/DailyMail test set, HeterSUMGraph (HSG) scores 42.31 R-1, 19.51 R-2 and 38.74 R-L.
  • On Multi-News, HeterDocSUMGraph (HDSG) reaches 46.05 / 16.35 / 42.08, ahead of Hi-MAP (45.21 / 16.29 / 41.39) and of single-graph HSG (45.66 / 16.22 / 41.80).
  • Trigram blocking lowers scores on Multi-News, since its human summaries are written as a whole and reuse key phrases.
  • In ablations, removing residual connections or sentence updating cuts R-1 to about 41.6. Edge features, word updating and BiLSTM initialization each also help.
  • The gain over Ext-BiLSTM grows with the average degree of word nodes, i.e. for more redundant documents. Code is released at github.com/brxx122/HeterSUMGraph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.553

@inproceedings{wang2020heterogeneous,
  title = {{Heterogeneous Graph Neural Networks for Extractive Document Summarization}},
  author = {Wang, Danqing and Liu, Pengfei and Zheng, Yining and Qiu, Xipeng and Huang, Xuanjing},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.553},
  url = {https://doi.org/10.18653/v1/2020.acl-main.553}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kristjan Arumae and Fei Liu. 2018. Reinforced extractive summarization with question-focused rewards. In Proceedings of ACL 2018, Student Research Workshop, pages 105–111. find
  2. Jaime G Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In SIGIR, volume 98, pages 335–336. find
  3. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 484–494. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  5. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739–3748. find
  6. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. arXiv preprint arXiv:1603.08887. find our card
  7. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479. find
  8. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy.… find our card
  9. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. find our card
  10. Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In ICML, pages 1263–1272. find our card
  11. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692. find our card
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  13. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  14. Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131–4141. find
  15. Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. find
  16. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  17. Hu Linmei, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. 2019. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… find
  18. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. Proceedings of the 6th International Conference on Learning Representations. find our card
  19. Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070– 5081. find our card
  20. Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721–3731, Hong Kong, China. Asso… find
  21. Yang Liu, Ivan Titov, and Mirella Lapata. 2019. Single document summarization as tree induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1745–1755. find
  22. Ling Luo, Xiang Ao, Yan Song, Feiyang Pan, Min Yang, and Qing He. 2019. Reading like her: Human reading inspired extractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  23. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. find
  24. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence. find our card
  25. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ça glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. find
  26. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers… find our card
  27. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. find our card
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  29. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  30. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1073–1083. find
  31. Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. 2016. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering, 29(1):17–37. find
  32. Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou. 2019. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. arXiv preprint arXiv:1905.07374. find
  33. Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu, and Xuan-Jing Huang. 2019b. A closer look at data bias in neural extractive summarization models. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 80–89. find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  35. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 654–6… find our card
  36. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903. find our card
  37. Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019a. Exploring domain shift in extractive text summarization. arXiv preprint arXiv:1908.11664. — [arXiv:1908.11664](https://arxiv.org/abs/1908.11664) find
  38. Hsiu-Yi Wang, Jia-Wei Chang, and Jen-Wei Huang. 2019b. User intention-based document summarization on heterogeneous sentence networks. In International Conference on Database Systems for Advanced Applications, pages 572–587. Springer. find
  39. Yang Wei. 2012. Document summarization method based on heterogeneous graph. In 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery, pages 1285–1289. IEEE. find
  40. A In order to select the best iteration number for H ET -ER SUMG RAPH , we compare performances of different t on the validation set of CNN/DM. All models are trained on a single GeForce RTX 2080 Ti GPU for about 5 epochs. As Table 5 shows, our H ETER SUMG RAPH has comparable results for t = 1 and t… find
  41. Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. arXiv preprint arXiv:1902.00863. find
  42. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. arXiv preprint arXiv:1910.14142. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2020-heterogeneous-graph-neural-networks-extractive/