Elamur Research⌕
☰
home / papers / 2021 / Extractive Summarization Considering Discourse and Coreference Relations based on Heterogeneous Graph

Extractive Summarization Considering Discourse and Coreference Relations based on Heterogeneous Graph

Summary

Extractive summarizers need to capture how parts of a document relate to each other. These relations exist at different levels of granularity: discourse links connect elementary discourse units (EDUs), and coreference links connect mentions of the same entity. Earlier graph-based summarizers mostly used homogeneous graphs with a single node type, which can represent only one kind of relation at a time.

Huang and Kurohashi instead build a heterogeneous document graph with three node types: sentences, EDUs and entities.

  • Discourse: directed dependency edges derived from the RST tree join EDUs.
  • Coreference: each entity cluster is linked to every EDU that mentions it.
  • Structure: each sentence is attached to its own EDUs.

Node vectors come from a pretrained encoder with attention pooling. Three rounds of graph attention (GAT) refine them, and a sigmoid layer then decides which EDUs to extract.

On CNN/DailyMail, the model outperforms BertSum at both sentence and EDU level. It ties with DiscoBERT on ROUGE-1/2 and beats it on ROUGE-L, without DiscoBERT's hand-crafted discourse-consistency rules. The study suggests that combining several linguistic relation types in one graph improves extraction.

Key points

  • Data: CNN/DailyMail (non-anonymized), with 287,227 / 13,368 / 11,490 documents for train / validation / test. Oracle labels were built with a ROUGE-based greedy heuristic, selecting at most 5 EDUs per document.
  • Preprocessing: Stanford CoreNLP for sentence splitting, the Ji and Eisenstein (2014) RST parser for segmentation and parsing, and a SpanBERT-based version of the Lee et al. (2017) coreference resolver.
  • Main results (R-1/R-2/R-L): the proposed model scores 43.61/20.81/41.12. BertSum (EDU) scores 42.73/20.03/40.16, and DiscoBERT scores 43.77/20.85/40.67.
  • Ablation:
    • Dropping discourse edges causes the largest loss (−0.19 R-1, −0.18 R-L).
    • Dropping coreference edges costs about −0.10.
    • Sentence–EDU links matter little (−0.04 R-1).
  • Setup: Longformer base encoder, input truncated to 1024 BPEs, 3 GAT iterations with 8 heads of size 64, and 50k training steps. Reported scores average the 3 best checkpoints.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Coreference resolution

Cited by

18 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.265

@inproceedings{huang2021extractive,
  title = {{Extractive Summarization Considering Discourse and Coreference Relations based on Heterogeneous Graph}},
  author = {Huang, Yin Jou and Kurohashi, Sadao},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.265},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.265}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. find
  2. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  3. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484–494, Berlin, Germany. Association for Computational Linguistics. find our card
  4. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. find our card
  5. Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163–1173, Atlanta, Georgia.… find
  6. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Y. Dong, J. Tang, S. Wu, J. Tian, N. V. Chawla, J. Rao, and H. Cao. 2012. Link prediction and recommendation across heterogeneous social networks. In 2012 IEEE 12th International Conference on Data Mining, pages 181–190. find
  9. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739–3748, Brussels, Belgium. Association for Computati… find
  10. Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res. (JAIR), 22:457– 479. find
  11. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28,… find our card
  12. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13–24, Baltimore, Maryland. Association for Computational Linguistics. find our card
  13. Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 137–147, Los Angeles. Association for Computational Linguistics. find
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  15. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Assoc… find
  16. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281. find our card
  17. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Ma… find our card
  18. R. Mihalcea and P. Tarau. 2004. TextRank: Bringing order into texts. In Proceedings of EMNLP-04and the 2004 Conference on Empirical Methods in Natural Language Processing. find
  19. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. find our card
  20. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… find
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  22. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations. find
  23. Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous graph neural networks for extractive document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6209–6219, Online. Association for Computational… find
  24. Y. Wei. 2012. Document summarization method based on heterogeneous graph. In 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery, pages 1285–1289. find
  25. Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. find
  26. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021–5031, Online. Association for Computational Linguistics. find our card
  27. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452–462, Vancouver, Canada. Association fo… find our card
  28. Xiao Yu, Quanquan Gu, Mianwei Zhou, and Jiawei Han. 2012. Citation Prediction in Heterogeneous Bibliographic Networks, pages 1119–1130. find
  29. Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HI-BERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059–5069, Florence, Italy. Association for Com… find
  30. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654– 663, Melbo… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2021-extractive-summarization-considering-discourse-coreference/