Elamur Research⌕
☰
home / papers / 2022 / RST Discourse Parsing with Second-Stage EDU-Level Pre-training

RST Discourse Parsing with Second-Stage EDU-Level Pre-training

Summary

Pre-trained language models such as XLNet have boosted RST discourse parsing. However, they are trained on sentence-level text, while RST parsers operate on elementary discourse units (EDUs). The authors address this mismatch with a second pre-training stage that continues training an existing PLM on EDU-level objectives. Large unlabeled corpora are automatically split into EDUs with a reimplemented segmenter, and two tasks are applied:

  • Next EDU prediction (NEP): decide whether two EDUs are adjacent.
  • Discourse marker prediction (DMP): recover a masked discourse marker at the start of an EDU.

For evaluation, they adapt a shift-reduce transition-based neural parser built on the transition system of Yu et al. (2018). The parser encodes pairs of neighbouring EDUs jointly (BiEDU representations) rather than isolated EDUs. On RST-DT, the combined pre-training raises F1 by 2.1 points, and the authors report state-of-the-art results. Notably, the pseudo-data needs only an EDU segmenter and a marker list, with no trained RST parser required. Gains do not carry over evenly to other domains.

Key points

  • NEP adapts BERT's next sentence prediction to EDU pairs, using balanced numbers of adjacent and non-adjacent pairs. DMP adapts masked language modeling by masking discourse markers, using the marker inventory of Fraser (2009).
  • The EDU segmenter used to prepare pre-training data is trained on RST-DT and reaches 96.0% F1.
  • Each NEP and DMP task helps on its own, and combining them gives the best result: +2.1 F1 on the RST-DT benchmark with gold EDUs as input.
  • Ablations confirm the value of discourse markers. Masking random words instead of markers slightly hurts performance, and DMP helps most on spans that contain markers.
  • On the GUM corpus, performance drops sharply out of domain, especially for conversation and vlog. The pre-training helps in academic, textbook and whow but slightly hurts bio, speech and vlog. Code is released as E-NNRSTParser on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.294

@inproceedings{yu2022rst,
  title = {{RST Discourse Parsing with Second-Stage EDU-Level Pre-training}},
  author = {Yu, Nan and Zhang, Meishan and Fu, Guohong and Zhang, Min},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.294},
  url = {https://doi.org/10.18653/v1/2022.acl-long.294}
}

References

51 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We proposed a second-stage EDU-level pre-training approach for PLM-based RST discourse parser, reducing the mismatch between the EDU-level RST discourse parsing and the pre-training of sentence-level contextualized language modeling. In addition, we extended our RST discourse parser with a light bi-… find
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
  3. Chloé Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual RST Discourse Parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304, Valencia, Spain. Association for Computational Linguistic… find our card
  4. Chloé Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Conference on Computational Linguistics (CoLing), pages 1903 – 1913, Osaka, Japan. find our card
  5. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue. find our card
  6. Alexis CONNEAU and Guillaume Lample. 2019. Cross-lingual Language Model Pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Vanessa Wei Feng and Graeme Hirst. 2012. Text-level Discourse Parsing with Rich Linguistic Features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 60–68, Jeju Island, Korea. Association for Computational Linguistics. find our card
  9. Vanessa Wei Feng and Graeme Hirst. 2014. A Linear-Time Bottom-Up Discourse Parser with Constraints and Post-Editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511–521, Baltimore, Maryland. Association for Computational L… find our card
  10. Bruce Fraser. 2009. An Account of Discourse Markers. International Review of Pragmatics, 1(2):293–320. Publisher: Brill. find
  11. Grigorii Guz and Giuseppe Carenini. 2020. Coreference for Discourse Parsing: A Neural Approach. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 160–167, Online. Association for Computational Linguistics. find our card
  12. Katsuhiko Hayashi, Tsutomu Hirao, and Masaaki Nagata. 2016. Empirical comparison of dependency conversions for RST discourse trees. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 128–136, Los Angeles. Association for Computational Linguistics… find
  13. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED AT-TENTION. find
  14. Michael Heilman and Kenji Sagae. 2015. Fast Rhetorical Structure Theory Discourse Parsing. arXiv:1505.02425 [cs]. ArXiv: 1505.02425. find our card
  15. Hugo Hernault, Helmut Prendinger, David A. du Verle, and Mitsuru Ishizuka. 2010. HILDA: A Discourse Parser Using Support Vector Machine Classification. dad, 1(3):1–33. find our card
  16. Patrick Huber and Giuseppe Carenini. 2019. Predicting Discourse Structure using Distant Supervision from Sentiment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), page… find
  17. Yangfeng Ji and Jacob Eisenstein. 2014. Representation Learning for Text-level Discourse Parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13–24, Baltimore, Maryland. Association for Computational Linguistics. find our card
  18. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  19. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining Intra- and Multi-sentential Rhetorical Parsing for Document-level Discourse Analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 486–496, Sofi… find
  20. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A Novel Discriminative Framework for Rhetorical Analysis. Computational Linguistics, 41(3):385–435. find our card
  21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. ICLR. find
  22. Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-Down RST Parsing Utilizing Granularity Levels in Documents. AAAI, 34(05):8099–8106. find
  23. Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2021. Improving Neural RST Parsing Model with Silver Agreement Subtrees. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  24. Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Top-down Discourse Parsing via Sequence Labelling. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 715–726, Online. Association for Computational Linguistics. find our card
  25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  26. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse Parsing with Attention-based Hierarchical Neural Networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas. Association for Computational Linguistics. find our card
  27. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and M Saiful Bari. 2019. A Unified Linear-Time Framework for Sentence-Level Discourse Parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4190– 4200, Florence, Italy. Association for Computational L… find our card
  28. Linlin Liu, Xiang Lin, Shafiq Joty, Simeng Han, and Lidong Bing. 2019a. Hierarchical Pointer Net Parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1007– 10… find
  29. Yang Liu and Mirella Lapata. 2017. Learning Contextually Informed Representations for Linear-Time Discourse Parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1289–1298, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692. find
  31. Zhengyuan Liu, Ke Shi, and Nancy Chen. 2020. Multilingual Neural RST Discourse Parsing. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6730–6738, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  32. Amandla Mabona, Laura Rimell, Stephen Clark, and Andreas Vlachos. 2019. Neural Generative Rhetorical Structure Parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),… find our card
  33. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark. Ass… find our card
  34. Philippe Muller, Chloé Braud, and Mathieu Morey. 2019. ToNy: Contextual embeddings for accurate multilingual discourse segmentation of full documents. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 115– 124, Minneapolis, MN. Association for Computational Lin… find
  35. Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. RST Parsing from Scratch. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1613–1625, Online. Association for Computational Li… find
  36. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… find our card
  37. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  38. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-I… find our card
  39. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67. find
  40. Deborah Schiffrin. 1987. Discourse Markers. Cambridge University Press. find
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  42. Wei Shi and Vera Demberg. 2019. Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM… find our card
  43. Radu Soricut and Daniel Marcu. 2003. Sentence Level Discourse Parsing using Syntactic and Lexical Information. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 228–235. find our card
  44. Longyin Zhang, Fang Kong, and Guodong Zhou. 2021. Adversarial Learning for Discourse Rhetorical Structure Parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pap… find our card
  45. Longyin Zhang, Yuqing Xing, Fang Kong, Peifeng Li, and Guodong Zhou. 2020. A Top-down Neural Architecture towards Text-level Parsing of Discourse Rhetorical Structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6386–6395, Online. Association for… find our card
  46. Mihai Surdeanu, Tom Hicks, and Marco Antonio Valenzuela-Escárcega. 2015. Two Practical Rhetorical Structure Theory Parsers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5, Denver, Colorado. Association f… find our card
  47. Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A Two-Stage Parsing Method for Text-Level Discourse Analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184–188, Vancouver, Canada. Association for Computational Linguist… find our card
  48. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… find our card
  49. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv:1906.08237 [cs]. ArXiv: 1906.08237. find our card
  50. Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based Neural RST Parsing with Implicit Syntax Features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559–570, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  51. Amir Zeldes. 2017. The GUM corpus: creating multilayer resources in the classroom. Lang Resources & Evaluation, 51(3):581–612. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2022-rst-discourse-parsing-second-stage/