Elamur Research⌕
☰
home / papers / 2022 / A Simple and Strong Baseline for End-to-End Neural RST-style Discourse Parsing

A Simple and Strong Baseline for End-to-End Neural RST-style Discourse Parsing

Summary

Progress in RST-style discourse parsing has been hard to judge. Recent neural parsers combine several factors at once, including parsing strategy, pre-trained encoder, ensembling, beam search and adversarial training, so it is unclear which one drives the gains. The authors build reference baselines from two simple open-source parsers: a top-down span-splitting parser (Kobayashi et al., 2020) and a bottom-up shift-reduce parser (Guz and Carenini, 2020). Both use small feed-forward networks over span vectors, where each span vector averages the encoder outputs for the span's first and last subwords. Each parser is paired with five transformer encoders: BERT, RoBERTa, XLNet, SpanBERT and DeBERTa. They are evaluated on RST-DT and Instr-DT.

The choice of language model matters far more than the choice of strategy. The bottom-up parser with DeBERTa beats the previous best systems. Further analysis shows that encoders pre-trained with span masking are especially helpful for long, cross-sentence spans. These results give future RST parsing work a dependable point of comparison. They also suggest that discourse-oriented pre-training is a promising research direction.

Key points

  • The bottom-up parser with DeBERTa reaches a fully labeled span F1 of 55.4 on RST-DT (standard Parseval). The prior best was 53.8, from Zhang et al. (2021) with XLNet.
  • Every encoder except BERT clearly improves both parsing strategies. Strategy differences are small compared with encoder differences.
  • Intra-sentential parsing shows little spread among the encoders. Multi-sentential parsing strongly favors DeBERTa and SpanBERT, which points to span masking as the key factor.
  • For N-N nuclearity, DeBERTa improves by more than 2 points on RST-DT and by more than 3 points on Instr-DT.
  • The code is released at github.com/nttcslab-nlp/RSTParser_EMNLP22. The authors note two limitations: weaker results on the small Instr-DT, and reliance on large pre-trained models that low-resource languages may lack.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.501 · arXiv 2210.08355

@inproceedings{kobayashi2022simple,
  title = {{A Simple and Strong Baseline for End-to-End Neural RST-style Discourse Parsing}},
  author = {Kobayashi, Naoki and Hirao, Tsutomu and Kamigaito, Hidetaka and Okumura, Manabu and Nagata, Masaaki},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.501},
  eprint = {2210.08355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.501}
}

References

42 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B (Methodological), 57(1):289–300. find
  2. Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212–2218, Lisbon, Portugal. Association for Computational Linguistics. find our card
  3. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 18–27, Vancouver. Association for Computational Linguistics. find our card
  4. David duVerle and Helmut Prendinger. 2009. A novel discourse parser based on support vector machine classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 665–673, Sunt… find our card
  5. William Falcon et al. 2019. Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning, 3. ↗
  6. Vanessa Wei Feng and Graeme Hirst. 2012. Text-level discourse parsing with rich linguistic features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 60–68, Jeju Island, Korea. Association for Computational Linguistics. find our card
  7. Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511– 521, Baltimore, Maryland. Association for Computational… find our card
  8. Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty, Steven C.H. Hoi, Caiming Xiong, Irwin King, and Michael Lyu. 2020. Discern: Discourse-aware entailment reasoning network for conversational machine reading. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing… find
  9. Grigorii Guz and Giuseppe Carenini. 2020. Coreference for discourse parsing: A neural approach. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 160–167, Online. Association for Computational Linguistics. find our card
  10. Grigorii Guz, Patrick Huber, and Giuseppe Carenini. 2020. Unleashing the power of neural discourse parsers - a context and structure aware approach using large scale pretraining. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3794–3805, Barcelona, Spain (Onli… find our card
  11. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the International Conference on Learning Representations. find our card
  12. Michael Heilman and Kenji Sagae. 2015. Fast rhetorical structure theory discourse parsing. CoRR, abs/1505.02425. find our card
  13. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13–24, Baltimore, Maryland. Association for Computational Linguistics. find our card
  14. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 486–496, Sofi… find
  15. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385–435. find our card
  16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361. find
  17. Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down rst parsing utilizing granularity levels in documents. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8099–8106. find
  18. Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2021. Improving neural RST parsing model with silver agreement subtrees. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  19. Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Top-down discourse parsing via sequence labelling. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 715–726, Online. Association for Computational Linguistics. find our card
  20. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069, Doha, Qatar. Association for Computational Linguistics. find our card
  21. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas. Association for Computational Linguistics. find our card
  22. Paul Pu Liang, Yao Chong Lim, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2019. Strong and simple baselines for multimodal utterance embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… find
  23. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and M Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4190– 4200, Florence, Italy. Association for Computational L… find our card
  24. Zhengyuan Liu and Nancy Chen. 2019. Exploiting discourse-level segmentation for extractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 116–121, Hong Kong, China. Association for Computational Linguistics. find
  25. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. find
  26. Fixing CoRR, Mary Ellen Okurowski Lynn Carlson, Daniel Marcu. 2002. RST Discourse Treebank. Philadelphia: Linguistic Data Consortium. find
  27. W.C. Mann and S.A Thompson. 1987. Rhetorical structure theory: A theory of text organization. Technical Report ISI/RS-87-190, USC/ISI. find
  28. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark. Ass… find our card
  29. Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. RST parsing from scratch. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1613–1625, Online. Association for Computational Li… find
  30. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… find our card
  31. Kenji Sagae and Alon Lavie. 2005. A classifier-based parser with linear run-time complexity. In Proceedings of the Ninth International Workshop on Parsing Technology, pages 125–132, Vancouver, British Columbia. Association for Computational Linguistics. find our card
  32. Ke Shi, Zhengyuan Liu, and Nancy F. Chen. 2020. An end-to-end document-level neural discourse parser exploiting multi-granularity representations. CoRR, abs/2012.11169. — [arXiv:2012.11169](https://arxiv.org/abs/2012.11169) find
  33. Rajen Subba and Barbara Di Eugenio. 2009. An effective discourse parser that uses rich linguistic information. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 566–574, Boulder, Colorado. A… find our card
  34. Jun Suzuki, Sho Takase, Hidetaka Kamigaito, Makoto Morishita, and Masaaki Nagata. 2018. An empirical study of building a strong baseline for constituency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 612–618, Melbo… find
  35. Longyin Zhang, Fang Kong, and Guodong Zhou. 2021. Adversarial learning for discourse rhetorical structure parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pap… find our card
  36. Longyin Zhang, Yuqing Xing, Fang Kong, Peifeng Li, and Guodong Zhou. 2020. A top-down neural architecture towards text-level parsing of discourse rhetorical structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6386–6395, Online. Association for… find our card
  37. Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184–188, Vancouver, Canada. Association for Computational Linguist… find our card
  38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  39. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021–5031, Online. Association for Computational Linguistics. find our card
  40. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find our card
  41. Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural RST parsing with implicit syntax features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559–570, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  42. Computing Infrastructure GeForce RTX 3090 Hyperparameters number of training epochs patience of early stopping batch size (\# of spans/actions) language model’s hidden size 768 (base) FFN’s hidden size dropout 0.2 learning rate scheduler Linear warm-up optimizer AdamW learning rate 2e-4, 1e-5 weight… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kobayashi-2022-simple-strong-baseline-end-end/