Elamur Research⌕
☰
home / papers / 2019 / Neural Generative Rhetorical Structure Parsing

Neural Generative Rhetorical Structure Parsing

Summary

Rhetorical Structure Theory (RST) parsing builds a tree over a document's elementary discourse units (EDUs), linking them with rhetorical relations and nuclearity. The main training resource, the English RST-DT, is small, and earlier parsers were all locally normalized discriminative models. These tend to suffer from label bias and to learn less efficiently from limited data. The authors propose the first generative RST parser, a document-level RNNG with bottom-up transitions. It has two actions: one generates an EDU, and one merges the top two stack units under a relation and nuclearity label. EDU generation is scored with a neural language model. Structural decisions use a stack representation that combines neural embeddings with hand-crafted structural features.

The authors show that the standard word-level beam search of Fried et al. (2017) pushes this kind of parser toward left-branching trees. They introduce a bag-level beam search that avoids this bias. With it, the generative model beats an otherwise identical discriminative baseline. It also matches state-of-the-art performance without extra training data, which suggests generative modelling suits low-resource discourse parsing.

Key points

  • Data and setup: experiments use RST-DT (347 training documents) with gold EDU segmentation, 18 coarse-grained relations and the nuclearity labels SN, NS and NN. Structural features come from sentence and paragraph boundaries and from spaCy dependency trees.
  • Search algorithm: on the development set, bag-level beam search improves on word-level beam search by 6.8 points of unlabelled F1 and 2.9 points of labelled F1. It wins at every beam size tested.
  • Generative vs. discriminative: on the test set, the generative model reaches a full-tree F1 of 45.0, compared with 42.4 for the discriminative baseline. That is a gain of 2.6, and it improves every metric by 1.9–2.7 points.
  • Comparison with prior parsers: test scores are S 67.1, N 57.4 and R 45.5. The N score is the best of all parsers in Morey et al. (2017)'s replication study. The full-tree score beats every published parser there that uses no additional training data.
  • Design findings: bottom-up traversal worked better than top-down, and conditioning only on the stack helped. Neural embeddings alone performed poorly unless structural features were added.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1233 · arXiv 1909.11049

@inproceedings{mabona2019neural,
  title = {{Neural Generative Rhetorical Structure Parsing}},
  author = {Mabona, Amandla and Rimell, Laura and Clark, Stephen and Vlachos, Andreas},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1233},
  eprint = {1909.11049},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1233}
}

References

48 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find our card
  2. Nicholas Asher. 2012. Reference to abstract objects in discourse, volume 50. Springer Science & Business Media. find
  3. Nicholas Asher, Nicholas Michael Asher, and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
  4. Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from rst discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212–2218. find our card
  5. Chlo Braud, Maximin Coavoux, and Anders Sgaard. 2017. Cross-lingual RST Discourse Parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304, Valencia, Spain. Association for Computational Linguistics. find our card
  6. Chlo Braud, Barbara Plank, and Anders Sgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1903–1913, Osaka, Japan. find our card
  7. The COLING 2016 Organizing Committee. find
  8. Jan Buys and Phil Blunsom. 2018. Neural Syntactic Generative Models with Exact Marginalization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 942–952, New Orleans, Louis… find
  9. Lynn Carlson and Daniel Marcu. 2001. Discourse tagging reference manual. ISI Technical Report ISI-TR-545, 54:56. find
  10. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural… find our card
  11. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent Neural Network Grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209, San Diego, California. Assoc… find
  12. Vanessa Wei Feng and Graeme Hirst. 2014a. A Linear-Time Bottom-Up Discourse Parser with Constraints and Post-Editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511– 521, Baltimore, Maryland. Association for Computational… find our card
  13. Vanessa Wei Feng and Graeme Hirst. 2014b. Patterns of local discourse coherence as a feature for authorship attribution. Literary and Linguistic Computing, 29(2):191–198. find
  14. Elisa Ferracane, Su Wang, and Raymond Mooney. 2017. Leveraging Discourse Information Effectively for Authorship Attribution. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 584–593, Taipei, Taiwan. Asian Federation of Natural… find
  15. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–166. find
  16. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520. IEEE. find
  17. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. find our card
  18. Katsuhiko Hayashi, Tsutomu Hirao, and Masaaki Nagata. 2016. Empirical comparison of dependency conversions for RST discourse trees. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 128– 136, Los Angeles. Association for Computational Linguistic… find
  19. Eugene Charniak et al. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336. find
  20. Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  21. Yangfeng Ji and Jacob Eisenstein. 2014. Representation Learning for Text-level Discourse Parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13–24, Baltimore, Maryland. Association for Computational Linguistics. find our card
  22. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In Proceedings of NAACL-HLT, pages 332–342. find
  23. Yangfeng Ji and Noah A Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 996–1005. find
  24. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A Novel Discriminative Framework for Rhetorical Analysis. Computational Linguistics, 41(3):385–435. find our card
  25. William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  27. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark. Ass… find our card
  28. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs]. ArXiv: 1412.6980. find
  29. Andrew Y. Ng and Michael I. Jordan. 2002. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Advances in neural information processing systems, pages 841–848. find
  30. Lingpeng Kong, Gabor Melis, Wang Ling, Lei Yu, and Dani Yogatama. 2019. Variational Smoothing in Recurrent Neural Network Language Models. arXiv:1901.09296 [cs]. ArXiv: 1901.09296. — [arXiv:1901.09296](https://arxiv.org/abs/1901.09296) find
  31. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. find
  32. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What Do Recurrent Neural Network Grammars Learn About Syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers… find
  33. Livia Polanyi. 1988. A formal model of the structure of discourse. Journal of pragmatics, 12(5-6):601–638. find
  34. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  35. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. Citeseer. find our card
  36. Geoffrey Sampson. 1997. Depth in English grammar. Journal of Linguistics, 33(1):131–151. find
  37. Radu Soricut and Daniel Marcu. 2003. Sentence Level Discourse Parsing using Syntactic and Lexical Information. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 228–235. find our card
  38. John D Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 282–289. Morgan Kaufmann Publishers Inc. find
  39. Mitchell Stern, Daniel Fried, and Dan Klein. 2017. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695–1700. find
  40. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse Parsing with Attention-based Hierarchical Neural Networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas. Association for Computational Linguistics. find our card
  41. Mihai Surdeanu, Tom Hicks, and Marco Antonio Valenzuela-Escarcega. 2015. Two Practical Rhetorical Structure Theory Parsers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5, Denver, Colorado. Association f… find our card
  42. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. arXiv preprint arXiv:1905.05682. find our card
  43. Zhiyang Teng and Yue Zhang. 2017. Head-Lexicalized Bidirectional Tree LSTMs. Transactions of the Association for Computational Linguistics, 5:163–177. find
  44. Bonnie L Webber. 1988. Discourse deixis and discourse processing. Discourse, 9. find
  45. D Yogatama, C Dyer, W Ling, and P Blunsom. 2017. Generative and discriminative text classification with recurrent neural networks. In Thirty-fourth International Conference on Machine Learning (ICML 2017). International Machine Learning Society. find
  46. Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural rst parsing with implicit syntax features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559–570. find
  47. Longyin Zhang, Cheng Sun, Xin Tan, and Fang Kong. 2018. Rst discourse parsing with tree-structured neural networks. In China Workshop on Machine Translation, pages 15–26. Springer. find
  48. Yue Zhang and Stephen Clark. 2008. A Tale of Two Parsers: Investigating and Combining Graph-based and Transition-based Dependency Parsing Using Beam-search. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 562–571, Stroudsburg, PA, USA. Associati… find our card

↑ back to top

This page in Markdown →