Elamur Research⌕
☰
home / papers / 2019 / Sentence Centrality Revisited for Unsupervised Summarization

Sentence Centrality Revisited for Unsupervised Summarization

Paper · 2019 · Hao Zheng and Mirella Lapata

Summary

Neural extractive summarizers rely on large collections of document–summary pairs. Such data rarely exist for every summary style, domain or language. Zheng and Lapata therefore return to unsupervised, graph-based sentence ranking in the spirit of TextRank and change how a sentence's centrality is scored.

They make two changes. First, sentence similarity comes from BERT, fine-tuned with a sentence-level distributional objective, rather than tf-idf-style overlap. Second, the graph's edges become directed: each similarity link is split into a forward-looking and a backward-looking edge with different weights. This uses position as a cheap stand-in for RST nuclearity, since earlier sentences in news tend to carry the core content. The resulting system, PacSum, ranks sentences by this weighted degree centrality.

It was tested on three news datasets: CNN+DM, NYT and the Chinese TTNews. PacSum beats strong unsupervised baselines and the LEAD baseline. Its results nearly match those of supervised models that learn from several hundred thousand training examples. This makes it a practical, low-data reference point for languages and domains lacking training corpora.

Key points

  • The datasets used are CNN+DM (11,490 docs), NYT (4,375) and TTNews (2,000, Chinese). The code is released as PacSum on GitHub (mswellhao/PacSum).
  • The forward and backward edge weights (λ1, λ2, constrained to sum to 1) are tuned on a small validation set. The best λ1 is usually negative, meaning similarity to preceding sentences lowers a sentence's centrality.
  • Plain degree centrality was used instead of PageRank, which gave only marginal differences.
  • A question-answering evaluation tested whether readers could answer questions from the summaries alone:
    • CNN/DM: 20 docs, 71 questions
    • NYT: 18 docs, 59 questions
    • TTNews: 50 docs, 50 questions
    • Five participants answered per summary.
  • In that QA study, PacSum (BERT) significantly outperformed LEAD-3 and was statistically on par with the supervised REFRESH. It remained below ORACLE, which itself scored under 100.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 159 works

Cite

Cite as Hao Zheng and Mirella Lapata. 2019. Sentence Centrality Revisited for Unsupervised Summarization. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1628

@inproceedings{zheng2019sentence,
  title = {{Sentence Centrality Revisited for Unsupervised Summarization}},
  author = {Hao Zheng and Mirella Lapata},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1628},
  url = {https://doi.org/10.18653/v1/p19-1628}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. 2016. Variations of the similarity function of TextRank for automated summarization. arXiv preprint arXiv:1602.03606. — [arXiv:1602.03606](https://arxiv.org/abs/1602.03606) find
  2. Sergey Brin and Michael Page. 1998. Anatomy of a large-scale hypertextual Web search engine. In Proceedings of the 7th Conference on World Wide Web, pages 107–117, Brisbane, Australia. find
  3. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484–494, Berlin, Germany.
  4. Eric Chu and Peter J. Liu. 2018. Unsupervised neural multi-document abstractive summarization. CoRR, abs/1810.05739. — [arXiv:1810.05739](https://arxiv.org/abs/1810.05739) find
  5. James Clarke and Mirella Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36(3):411–441. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  7. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1998–2008, Berlin, Germany…
  8. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. find
  9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015a. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28,…
  10. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015b. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693– 1701. Morgan, Kaufmann.
  11. Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihito Yasuda, and Masaaki Nagata. 2013. Single-document summarization as a tree knapsack problem. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1515–1520, Seattle, Washington, USA. find
  12. Lifeng Hua, Xiaojun Wan, and Lei Li. 2017. Overview of the nlpcc 2017 shared task: Single document summarization. In National CCF Conference on Natural Language Processing and Chinese Computing, pages 942–947. Springer. find
  13. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 486–496, Sofi…
  14. Cristobald de Kerchove and Paul Van Dooren. 2008. The pagetrust algorithm: How to rank web pages when negative links are allowed? In Proceedings of the 2008 SIAM International Conference on Data Mining, pages 346–352. SIAM. find
  15. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  16. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 3294–3302. Curran A… find
  17. Piji Li, Zihao Wang, Wai Lam, Zhaochun Ren, and Lidong Bing. 2017. Salience estimation via variational auto-encoders for multi-document summarization. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3497–3503, San Francisco, California. find
  18. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas.
  19. Chin-Yew Lin and Eduard Hovy. 1997. Identifying topics by position. In Proceedings of the 5th Conference on Applied Natural Language Processing, pages 283–290, Washington, DC, USA. find
  20. Chin-Yew Lin and Eduard Hovy. 2002. From single to multi-document summarization: A prototype system and its evaluation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 457–464, Pennsylvania, Philadelphia. find
  21. Chin Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 71–78, Edmonton, Canada.
  22. Yang Liu and Mirella Lapata. 2017. Learning contextually informed representations for linear-time discourse parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1300–1309, Copenhagen, Denmark.
  23. Shulei Ma, Zhi-Hong Deng, and Yunlun Yang. 2016. An unsupervised multi-document summarization framework based on neural document model. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1514–1523, Osaka, Japan. find
  24. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281.
  25. Daniel Marcu. 1997. From discourse structures to text summaries. In Proceedings of the ACL Workshop on Intelligent Scalable Text Summarization, pages 82– 88, Madrid, Spain. find
  26. Daniel Marcu. 1998. Improving summarization through rhetorical parsing tuning. In Proceedings of the 6th Workshop on Very Large Corpora, pages 206–215, Montréal, Canada. find
  27. Rada Mihalcea. 2004. Graph-based ranking algorithms for sentence extraction, applied to text summarization. In The Companion Volume to the Proceedings of 42st Annual Meeting of the Association for Computational Linguistics, pages 170–173, Barcelona, Spain. find
  28. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into texts. In Proceedings of EMNLP 2004, pages 404–411, Barcelona, Spain. find
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pa…
  30. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3075–3081, San Francisco, California.
  31. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. find
  32. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. find
  33. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pape…
  34. Ani Nenkova. 2005. Automatic text summarization of newswire: Lessons learned from the document understanding conference. In Proceedings of the 20th National Conference on Artificial Intelligence, pages 1436–1441, Pittsburgh, Pennsylvania. find
  35. Ani Nenkova, Kathleen McKeown, et al. 2011. Automatic summarization. Foundations and Trends R in Information Retrieval, 5(2–3):103–233. find
  36. You Ouyang, Wenjie Li, Qin Lu, and Renxian Zhang. 2010. A study on position information in document summarization. In Coling 2010: Posters, pages 919–927, Beijing, China. find
  37. Daraksha Parveen, Hans-Martin Ramsl, and Michael Strube. 2015. Topical coherence for graph-based extractive summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1949–1954, Lisbon, Portugal. find
  38. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada.
  39. Tamara Polajnar, Laura Rimell, and Stephen Clark. 2015. An exploration of discourse-based sentence spaces for compositional distributional semantics. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 1–11, Lisbon, Portuga… find
  40. Dragomir R. Radev, Hongyan Jing, and Malgorzata Budzikowska. 2000. Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In Proceedings of the NAACL-ANLP 2000 Workshop: Automatic Summarization, pages 21–30, Seattle, Washington.
  41. Xiaojun Wan and Jianwu Yang. 2008. Multi-document summarization using cluster-based link analysis. In Proceedings of the 31st Annual International ACL SIGIR Conference on Research and Development in Information Retrieval, pages 299–306, Singapore. find
  42. Wenpeng Yin and Yulong Pei. 2015. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, pages 1383–1389, Buenos Aires, Argentina. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2019-sentence-centrality-revisited-unsupervised-summarization/