Elamur Research⌕
☰
home / papers / 2021 / Using and comparing Rhetorical Structure Theory parsers with rst-workbench

Using and comparing Rhetorical Structure Theory parsers with rst-workbench

Paper · 2021 · Arne Neumann

Summary

RST discourse parsing is hampered by small hand-annotated corpora, weak evaluation practice, and parsers that are hard to install and produce incompatible outputs. The author argues that annotators could build RST corpora faster by choosing the best of several automatic analyses and correcting it, much as translators post-edit machine output. To support this, the paper presents rst-workbench, a system demonstration focused on usability rather than benchmark scores. It packages six open-source end-to-end RST parsers as Docker containers and gives each one a small REST wrapper. A browser front-end sends the input text to all parsers in parallel and shows each tree as soon as it is ready. A conversion service built on discoursegraphs turns the parsers' different outputs into rs3, so any result can be opened, edited and saved in rstWeb. The tool makes side-by-side comparison of RST parsers practical. It also lowers the cost of machine-assisted annotation and lets the parsers be called as web services inside other NLP pipelines.

Key points

  • The bundled parsers are HILDA, Feng and Hirst (2014), DPLP, Heilman and Sagae (2015), CODRA and StageDP. Docker images on hub.docker.com/u/nlpbox cover all of them except HILDA, whose license blocks redistribution; only its Dockerfile is provided.
  • Each parser gets a minimal REST API written with the Python hug library. As a result, parsers can run on separate machines and new parsers are easy to add.
  • Output formats are converted to rs3 (corpora otherwise use dis or rs3) via discoursegraphs. The author added a REST interface to rstWeb so that it can visualize and post-edit the results.
  • Running all parsers on one text can take up to about two minutes, mostly spent loading syntax-parser models. Moving the syntax parser into its own service, already done for DPLP, reduces this.
  • Background: Morey et al. (2017) found that most reported RST parser gains from 2013–2017 come from differences in how Marcu's evaluation was implemented. Rare-relation performance is also seldom compared.

Interesting ideas

  • I1. Run several independent parsers on the same text, let a human pick the most plausible tree, and correct only that one; this turns annotation into cheap post-editing.
  • I2. Convert every tool's native output through one central hub into a single shared format, so visualization and editing tools are written only once.
  • I3. Wrap each heavyweight tool in a thin service with a uniform request/response contract; the front-end then only needs each tool's address and output format.
  • I4. Most of the latency came from repeatedly loading syntax models, so keep those models resident as a long-running service.
  • I5. Reported gains can be artifacts of differing metric implementations, and averaged scores hide how rare labels fare.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): The witness-parser silver pipeline could show disagreeing trees side by side, so a human or LLM teacher corrects the best candidate instead of annotating from scratch.
  • I2 (tree): Treebank and UD dialect conversion could go through one canonical internal format, with one converter per source rather than pairwise converters.
  • I3 (learn): Witness parsers and teacher tiers could sit behind one interface, so adding or swapping a witness needs no pipeline changes.
  • I4 (tree): gparse and the perceptron parser could run as persistent services shared by later stages, avoiding repeated GPU and model start-up.
  • I5 (learn): Fix one reference scorer for all evaluations and report per-label results for rare relations or frames next to the overall averages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Discourse analysis · 394 works

Cite

Cite as Arne Neumann. 2021. Using and comparing Rhetorical Structure Theory parsers with rst-workbench. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-demos.1

@inproceedings{neumann2021using,
  title = {{Using and comparing Rhetorical Structure Theory parsers with rst-workbench}},
  author = {Arne Neumann},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-demos.1},
  url = {https://doi.org/10.18653/v1/2021.eacl-demos.1}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Heilman and Kenji Sagae. 2015. Fast Rhetorical Structure Theory Discourse Parsing. arXiv preprint arXiv:1505.02425. ↗
  2. Mohamed Abdalla, Frank Rudzicz, and Graeme Hirst. 2018. Rhetorical structure and Alzheimers disease. Aphasiology, 32(1). ↗
  3. Philipp Helfrich, Elias Rieb, Giuseppe Abrami, Andy Lücking, and Alexander Mehler. 2018. TreeAnnotator: Versatile Visual Annotation of Hierarchical Text Relations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA). find
  4. Chloé Braud, Maximin Coavoux, and Anders Søgaard. 2017a. Cross-lingual RST Discourse Parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304, Valencia, Spain. Association for Computational Linguistics. ↗
  5. Hugo Hernault, Helmut Prendinger, David A. duVerle, and Mitsuru Ishizuka. 2010. HILDA: A Discourse Parser Using Support Vector Machine Classification. Dialogue & Discourse, 1(3).
  6. Chloé Braud, Ophélie Lacroix, and Anders Søgaard. 2017b. Cross-lingual and cross-domain discourse segmentation of entire documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 237–243, Vancouver, Canada. Association for Computational Linguistics. find
  7. Patrick Huber and Giuseppe Carenini. 2020. MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7442–7457, Online. Association for Computational Linguistics.
  8. Chloé Braud, Ophélie Lacroix, and Anders Søgaard. 2017c. Does syntax help discourse segmentation? Not so much. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2432–2442, Copenhagen, Denmark. Association for Computational Linguistics. find
  9. Yangfeng Ji and Jacob Eisenstein. 2014. Representation Learning for Text-level Discourse Parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13–24, Baltimore, Maryland. Association for Computational Linguistics.
  10. Iria da Cunha, Eric SanJuan, Juan-Manuel Torres-Moreno, M. Teresa Cabré, and Gerardo Sierra. 2012. A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-sentence Discourse Segments in Spanish. In Proceedings of the 13th International Conference in Computational Linguistics and Intelligent Text Processing (CICLing 2012). find
  11. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A Novel Discriminative Framework for Rhetorical Analysis. Computational Linguistics, 41(3):385–435. ↗
  12. Ioannis Konstas and Mirella Lapata. 2013. Inducing Document Plans for Concept-to-Text Generation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1503–1514, Seattle, Washington, USA. Association for Computational Linguistics. find
  13. Vanessa Wei Feng and Graeme Hirst. 2014. A Linear-Time Bottom-Up Discourse Parser with Constraints and Post-Editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511– 521, Baltimore, Maryland. Association for Computational Linguistics.
  14. Maarit Koponen. 2016. Is machine translation post-editing worth the effort? A survey of research into post-editing and effort. The Journal of Specialised Translation, 25:131–148. find
  15. Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. thesis, University of California, Irvine. find
  16. Thomas Krause and Amir Zeldes. 2014. ANNIS3: A new architecture for generic corpus query and visualization. Literary and Linguistic Computing. ↗
  17. Federico Gaspari, Antonio Toral, Sudip Kumar Naskar, Declan Groves, and Andy Way. 2014. Perception vs Reality: Measuring Machine Translation Post-Editing Productivity. In Proc. Third Workshop on Post-Editing Technology and Practice. find
  18. Daniel Marcu. 1997. The Rhetorical Parsing, Summarization, and Generation of Natural Language Text. Ph.D. thesis, Department of Computer Science. University of Toronto. 7 See the Dockerfile in https://github.com/ NLPbox/dplp-docker. ↗
  19. Daniel Marcu. 2000. The rhetorical parsing of unrestricted texts: a surface-based approach. Computational Linguistics, 26(3):395–448. the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5, Denver, Colorado. Association for Computational Linguistics. ↗
  20. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  21. Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A Two-Stage Parsing Method for Text-Level Discourse Analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184–188, Vancouver, Canada. Association for Computational Linguistics. ↗
  22. Arne Neumann. 2015. discoursegraphs: A graph-based merging tool and converter for multilayer annotated corpora. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015), pages 309–312, Vilnius, Lithuania. Linköping University Electronic Press, Sweden. find
  23. Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward Fast and Accurate Neural Discourse Segmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 962–967, Brussels, Belgium. Association for Computational Linguistics. ↗
  24. Arne Neumann. 2016. Merging and validating heterogenous, multi-layered corpora with discoursegraphs. Journal for Language Technology and Computational Linguistics, 31(1):101–115. ↗
  25. Amir Zeldes. 2016. rstWeb - A Browser-based Annotation Interface for Rhetorical Structure Theory and Discourse Relations. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5, San Diego, California. Association for Computational Linguistics. ↗
  26. Michael O’Donnell. 2000. RSTTool 2.4 - A markup Tool for Rhetorical Structure Theory. In INLG’2000 Proceedings of the First International Conference on Natural Language Generation, pages 253–256, Mitzpe Ramon, Israel. Association for Computational Linguistics.
  27. Hongxin Zhang and Haitao Liu. 2016. Quantitative Aspects of RST Rhetorical Relations across Individual Levels. Glottometrics, 33:8–24. find
  28. Thiago Alexandre Salgueiro Pardo and Maria das Graças Volpe Nunes. 2008. On the Development and Evaluation of a Brazilian Portuguese Discourse Parser. RITA, 15(2):43–64. ↗
  29. Anayeli Paulino, Gerardo Sierra, Laura Hernández-Domı́nguez, Iria da Cunha, and Gemma Bel-Enguix. 2018. Rhetorical relations in the speech of Alzheimers patients and healthy elderly subjects: An approach from the RST. Computación y Sistemas, 22(3). ↗
  30. David Reitter. 2003. Simple Signals for Complex Rhetorics: On Rhetorical Analysis with Rich-Feature Support Vector Models. LDV Forum, 18(1). ↗
  31. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A Systematic Study of Neural Discourse Models for Implicit Discourse Relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 281–291, Valencia, Spain. Association for Computational Linguistics. ↗
  32. K. Sumita, K. Ono, T. Chino, T. Ukita, and S. Amano. 1992. A Discourse Structure Analyzer for Japanese Text. In Proceedings International Conference on Fifth Generation Computer Systems, pages 1133– 1140. find
  33. Mihai Surdeanu, Tom Hicks, and Marco Antonio Valenzuela-Escárcega. 2015. Two Practical Rhetorical Structure Theory Parsers. In Proceedings of

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neumann-2021-comparing-rhetorical-structure-theory-parsers/