Elamur Research⌕
☰
home / papers / 2023 / What's Hard in English RST Parsing? Predictive Models for Error Analysis

What's Hard in English RST Parsing? Predictive Models for Error Analysis

Paper · 2023 · Yang Janet Liu, Tatsuya Aoyama and Amir Zeldes

Summary

English RST discourse parsers built on neural models still perform poorly, especially outside their training domain, and it has been unclear which properties of a text cause their mistakes. The authors test several candidate explanations together:

  • whether a relation is explicitly signalled by a connective,
  • whether a misleading connective is present,
  • whether a relation crosses sentence boundaries,
  • unit length, genre, and the share of unseen words.

To do this, they hand-annotated the RST-DT test set and the GUM dev/test sets with PDTB-style discourse markers. Each marker token is either tied to the gold relation it signals or labelled a distractor. They then ran one bottom-up and one top-down SOTA parser, compared gold and predicted trees at the EDU level, and trained XGBoost classifiers to predict where each parser goes wrong. These classifiers reach 76.3% and 76.6% accuracy.

The strongest predictors are inter-sentential status and the specific relation. Explicitness matters less, OOV rate matters little, genre stays influential, and the two architectures fail in similar places. Distractors often reflect real competing relations, which suggests that RST's single-tree constraint mixes some valid alternative readings in with errors.

Key points

  • New annotation of discourse markers (strictly following the PDTB English inventory) on the RST-DT test set and the GUM v9 dev/test sets. Each marker is linked to a gold RST relation or marked as a distractor, and annotation was done in rstWeb.
  • Inter-annotator agreement on the 38 RST-DT test documents was high: F1 of 95.2 for marker detection and 96.7 for the associated relation given a marker.
  • The error-prediction models predict the bottom-up parser's errors with 76.3% accuracy and the top-down parser's with 76.6%. In order of importance, the features are:
    • inter-sentential status and the relation,
    • EDU length,
    • genre,
    • OOV rate.
  • Of the 174 distractors in GUM, about 62% still led to errors. In 68.5% (bottom-up) and 63.6% (top-down) of those errors, the majority predicted label matched the relation the distractor would normally signal.
  • In RST-DT's double-annotated subset, 9 of 12 distractors corresponded to the primary relation chosen by the second annotator. Code and data are at github.com/janetlauyeung/NLPErrors4RST.

Interesting ideas

  • I1. Instead of testing one suspected cause of errors at a time, train a gradient-boosted classifier that predicts per unit whether the system errs, then read off feature importance so that competing explanations are weighed jointly and their interactions are accounted for.
  • I2. Annotate not only the cues that support the gold label but also "distractor" cues that point toward a different label, then check whether wrong predictions follow the distractor; this separates genuine confusion from misleading surface signals.
  • I3. Before counting a mismatch as an error, compare it with an independent second annotation, because some "errors" turn out to be valid alternative analyses that a strict single-structure scheme cannot express.
  • I4. A model that predicts where the parser will fail can guide annotation: it can highlight low-confidence regions, warn annotators, and prioritise documents for manual labelling in an active-learning loop.
  • I5. Judging errors on the completed gold versus predicted structure, unit by unit after converting trees to dependencies, gives an output-centric view that ties each mistake to the properties of a concrete unit.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Train a small XGBoost-style predictor over gparse and perceptron outputs with features such as attachment distance, clause boundary crossing, token length, OOV and genre, to rank what actually drives LAS loss.
  • I2 (frames): Build a test slice where connectives like "but" or "if" do not carry their usual relation, and check whether prag's relation and speech-act decisions are being hijacked by those words.
  • I3 (learn): When witness parsers disagree with silver data, flag disagreements that another witness also supports as possible concurrent readings, so they are not fed into repair rules as plain mistakes.
  • I4 (learn): Use a failure predictor to choose which documents go to the LLM teacher tiers or to human checking first, spending supervision where errors are most likely.
  • I5 (answer): Attribute wrong answers on fables and school tests to specific sentences or clauses and their properties, rather than scoring only at the document level.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States88 works · 4,732 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDiscourse analysis · 466 works

Cite

Cite as Yang Janet Liu, Tatsuya Aoyama and Amir Zeldes. 2023. What's Hard in English RST Parsing? Predictive Models for Error Analysis. Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.sigdial-1.3

@inproceedings{liu2023what,
  title = {{What's Hard in English RST Parsing? Predictive Models for Error Analysis}},
  author = {Yang Janet Liu, Tatsuya Aoyama and Amir Zeldes},
  booktitle = {Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2023},
  doi = {10.18653/v1/2023.sigdial-1.3},
  url = {https://doi.org/10.18653/v1/2023.sigdial-1.3}
}

References

46 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 141–151, New Orleans, Louisiana. Association for Computational Linguistics. find
  2. Tatsuya Aoyama, Shabnam Behzad, Luke Gessler, Lauren Levine, Jessica Lin, Yang Janet Liu, Siyao Peng, Yilun Zhu, and Amir Zeldes. 2023. GENTLE: A genre-diverse multilayer challenge set for English NLP and linguistic evaluation. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 166–178, Toronto, Canada. Association for Computational Linguistics. ↗
  3. Debopam Das and Maite Taboada. 2014. RST Signalling Corpus Annotation Manual. Technical report, Simon Fraser University. find
  4. Nicholas Asher and Alex Lascarides. 2003. Logics of Conversation. Studies in Natural Language Processing. Cambridge University Press, Cambridge. find
  5. Debopam Das and Maite Taboada. 2017. Signalling of coherence relations in discourse, beyond discourse markers. Discourse Processes, 55(8):743–770. ↗
  6. Katherine Atwell, Junyi Jessy Li, and Malihe Alikhani. 2021. Where are we in discourse relation recognition? In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 314–325, Singapore and Online. Association for Computational Linguistics. ↗
  7. Debopam Das, Maite Taboada, and Paul McFetridge. 2019. RST Signalling Corpus. LDC2015T10. find
  8. Vera Demberg, Fatemeh Torabi Asr, and Merel Scholman. 2019. How compatible are our discourse annotation frameworks? insights from mapping RST-DT and PDTB annotations. Dialogue & Discourse, 10(1):87––135. ↗
  9. Katherine Atwell, Anthony Sicilia, Seong Jae Hwang, and Malihe Alikhani. 2022. The change that matters in discourse parsing: Estimating the impact of domain shift on parser error. In Findings of the Association for Computational Linguistics: ACL 2022, pages 824–845, Dublin, Ireland. Association for Computational Linguistics. ↗
  10. Luke Gessler, Lauren Levine, and Amir Zeldes. 2022. Midas loop: A prioritized human-in-the-loop annotation for large scale multilayer data. In Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pages 103–110, Marseille, France. European Language Resources Association. find
  11. Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, and Amir Zeldes. 2023. find
  12. The DISRPT 2023 shared task on elementary discourse unit segmentation, connective detection, and relation classification. In Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023), pages 1–21, Toronto, Canada. The Association for Computational Linguistics.
  13. Luke Gessler, Yang Liu, and Amir Zeldes. 2019. A discourse signal annotation system for RST trees. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 56–61, Minneapolis, MN. Association for Computational Linguistics.
  14. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory. In Current and New Directions in Discourse and Dialogue, Text, Speech and Language Technology 22, pages 85–112. Kluwer, Dordrecht. ↗
  15. Luke Gessler, Siyao Peng, Yang Liu, Yilun Zhu, Shabnam Behzad, and Amir Zeldes. 2020. AMALGUM – a free, balanced, multilayer English web corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5267–5275, Marseille, France. European Language Resources Association. ↗
  16. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA. Association for Computing Machinery. ↗
  17. Grigorii Guz and Giuseppe Carenini. 2020. Coreference for discourse parsing: A neural approach. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 160–167, Online. Association for Computational Linguistics. ↗
  18. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 486–496, Sofia, Bulgaria. Association for Computational Linguistics.
  19. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  20. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385–435. ↗
  21. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 154–164, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. find
  22. René Knaebel. 2021. discopy: A neural system for shallow discourse parsing. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 128–133, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. ↗
  23. William C Mann and Sandra A Thompson. 1988. Rhetorical Structure Theory: Toward a Functional Theory of Text Organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. ↗
  24. Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down RST parsing utilizing granularity levels in documents. In AAAI Conference on Artificial Intelligence. find
  25. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Special Issue on Using Large Corpora, Computational Linguistics, 19(2):313–330. ↗
  26. Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25–35, Baltimore, Maryland. Association for Computational Linguistics. ↗
  27. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  28. Jessica Lin and Amir Zeldes. 2021. WikiGUM: Exhaustive entity linking for wikification in 12 genres. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 170–175, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  29. Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. 2012. Constrained decoding for text-level discourse parsing. In Proceedings of COLING 2012, pages 1883–1900, Mumbai, India.
  30. The COL-ING 2012 Organizing Committee. find
  31. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and M Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4190–4200, Florence, Italy. Association for Computational Linguistics.
  32. Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. RST parsing from scratch. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1613–1625, Online. Association for Computational Linguistics.
  33. Yang Liu. 2019. Beyond the Wall Street Journal: Anchoring and comparing discourse signals across genres. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 72– 81, Minneapolis, MN. Association for Computational Linguistics.
  34. Noriki Nishida and Hideki Nakayama. 2020. Unsupervised discourse constituency parsing using Viterbi EM. Transactions of the Association for Computational Linguistics, 8:215–230. ↗
  35. Yang Liu and Mirella Lapata. 2017. Learning contextually informed representations for linear-time discourse parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1289–1298, Copenhagen, Denmark. Association for Computational Linguistics.
  36. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA).
  37. Yang Liu and Amir Zeldes. 2019. Discourse relations and signaling information: Anchoring discourse signals in RST-DT. Proceedings of the Society for Computation in Linguistics, 2(35):314–317. ↗
  38. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014. Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. ↗
  39. Yang Janet Liu and Amir Zeldes. 2023. Why can’t discourse parsing generalize? A thorough investigation of the impact of data diversity. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3104–3122, Dubrovnik, Croatia. Association for Computational Linguistics.
  40. Valentina Pyatkin, Frances Yung, Merel C. J. Scholman, Reut Tsarfaty, Ido Dagan, and Vera Demberg. 2023. Design choices for crowdsourcing implicit discourse relations: Revealing the biases introduced by task design. ↗
  41. Merel Scholman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, and Vera Demberg. 2022. Design choices in crowdsourcing discourse relation annotations: The effect of worker selection and training. ↗
  42. Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. DMRST: A joint framework for document-level multilingual RST discourse segmentation and parsing. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2148–2156, Marseille, France. European Language Resources Association. find
  43. Zheng Zhao and Bonnie Webber. 2021. Revisiting shallow discourse parsing in the PDTB-3: Handling intra-sentential implicits. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 107–121, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics.
  44. Erbo Shang, Xiaohua Liu, Hailong Wang, Yangfeng Rong, and Yuerong Liu. 2019. Research on the application of artificial intelligence and distributed parallel computing in archives classification. In 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), pages 1267–1271. find
  45. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021. OntoGUM: Evaluating contextualized SOTA coreference resolution on 12 more genres. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 461–467, Online. Association for Computational Linguistics. ↗
  46. Caroline Sporleder and Alex Lascarides. 2004. Combining hierarchical clustering and machine learning to predict high-level discourse structure. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 43–49, Geneva, Switzerland. COLING. A Validation Performance find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-what-s-hard-english-rst/