Elamur Research⌕
☰
home / papers / 2024 / Signals as Features: Predicting Error/Success in Rhetorical Structure Parsing

Signals as Features: Predicting Error/Success in Rhetorical Structure Parsing

Paper · 2024 · Martial Pastor and Nelleke Oostdijk

Summary

The paper asks which linguistic cues actually help or hinder a neural RST discourse parser when it labels coherence relations. The authors ran the pretrained DMRST parser (XLM-RoBERTa-base, multilingual, top-down) on the RST-DT test set with gold EDU segmentation. They then matched its outputs to the signal annotations of Das and Taboada's RST Signaling corpus, after correcting a token-offset bug that had kept the two resources from aligning. Each relation's signals were encoded as a binary vector. An XGBoost classifier learned from these vectors to predict whether the parser would label that relation correctly or wrongly, and feature importance was read from classification gain. The meta-model reached 0.78 accuracy. The analysis shows that discourse markers are frequent but unreliable and can mislead the parser. Syntactic, graphical and genre signals tend to go with correct labels. Relations whose signals are "unsure" or rely on lexical cues tend to go with errors. This matters because it gives an interpretable, feature-level account of where a black-box discourse parser succeeds or fails. The alignment code and data are public.

Key points

  • Parser setup and score: DMRST labelled the 2306 relations of the RST-DT test set (38 documents, 18 harmonized labels after Braud et al. 2017) with 0.67 accuracy under RST-Parseval, using gold EDUs.
  • Signal inventory: The RST Signaling corpus has 50 signals in three classes: single, combined and unsure. Single signals are split into eight types: numerical, genre, graphic, syntactic, semantic, lexical, reference and DM.
  • Meta-model accuracy: XGBoost on binary signal features predicted parser error or success with 0.78 accuracy on a held-out set of 761 relations.
  • Signals linked to success:
    • syntactic signals: 91.6% of cases were correctly predicted successes;
    • genre signals: about 83%;
    • graphical signals: 82.8%;
    • DMs: only 59.0%.
  • Where DMs work and fail: DMs help for CONTRAST ("but"), JOINT ("and") and CONDITION ("if"). They do poorly for BACKGROUND, CAUSE and TEMPORAL.
  • Signals linked to errors:
    • "unsure": 66 of 78 cases (84.6%) were correctly flagged as parser errors;
    • "indicative word": errors mostly on EVALUATION;
    • "lexical chain": errors on 20 of 37 cases.
  • Resource: Code and aligned data are released at github.com/metabolean5/signals-as-features.

Interesting ideas

  • I1. A meta-model of the parser's mistakes: Instead of only scoring a neural parser, train a small, interpretable classifier on human-annotated linguistic features to predict when the parser will be right or wrong, then read feature gains as a diagnosis.
  • I2. Missing cues as a warning sign: Cases where human annotators could find no clear signal ("unsure") are exactly where the model fails, so absence of cues is itself a strong error predictor.
  • I3. Distrust surface connectives: A connective like "when" or "since" can push a parser toward the wrong relation, while structural cues such as relative clauses, reported speech or interrupted clauses are more dependable.
  • I4. Fix the alignment before concluding a resource is unusable: An off-by-some token-position bug had made the signal annotations look unaligned, and repairing it unlocked the whole corpus.
  • I5. Analyse with gold segmentation: Using gold EDUs keeps segmentation noise out of the analysis of relation labelling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Train a gradient-boosted error predictor over explicit features of each arc (relation type, construction, punctuation, genre) to locate weak spots in gparse and steer where silver data or repair rules are needed.
  • I2 (frames): When prag assigns a discourse or causal relation without any detectable cue, mark it as low-confidence in the explanation trace rather than asserting it.
  • I3 (frames): Base relation decisions between clauses on UD structure (acl, ccomp with speech verbs, parataxis) first and use connectives only as a secondary vote, especially for temporal and causal relations.
  • I4 (tree): When converting or aligning treebanks and annotation layers, run offset sanity checks, since silent misalignment can make good data appear useless.
  • I5 (learn): Evaluate each pipeline stage on gold input from the previous stage alongside end-to-end runs, so a stage's own errors are not blamed on upstream tokenization or parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Radboud University NijmegenNijmegen · Netherlands20 works · 1,688 citations🎓Center for Language StudiesNijmegen · Netherlands3 works · 201 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.Discourse analysis · 466 works

Cite

Cite as Martial Pastor and Nelleke Oostdijk. 2024. Signals as Features: Predicting Error/Success in Rhetorical Structure Parsing. Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.codi-1.13

@inproceedings{pastor2024signals,
  title = {{Signals as Features: Predicting Error/Success in Rhetorical Structure Parsing}},
  author = {Martial Pastor and Nelleke Oostdijk},
  booktitle = {Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.codi-1.13},
  url = {https://doi.org/10.18653/v1/2024.codi-1.13}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Initially, the examination of imbalances in characteristics constituted a challenge because of the multilingual composition of the training dataset. Furthermore, depending on a single model checkpoint for experimentation introduces the potential for errors influenced by coincidental variations in training. Additionally, we highlight that the corpus is restricted to newswire data, and exploring data from different genres is likely to provide additional insights. It is also important to mention that in the current study, we specifically examined only those instances of potential signals that were identified as relevant for labeling coherence relations. This approach thus excluded what Liu et al. (2023) refer to as distractors. find
  2. Chloé Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual RST discourse parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304, Valencia, Spain. Association for Computational Linguistics. ↗
  3. Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, and Amir Zeldes. 2023. find
  4. The DISRPT 2023 shared task on elementary discourse unit segmentation, connective detection, and relation classification. In Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023), pages 1–21, Toronto, Canada. The Association for Computational Linguistics.
  5. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue. ↗
  6. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785– 794. ↗
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  8. Debopam Das. 2014. Signalling of coherence relations in discourse. Ph.D. thesis. find
  9. Debopam Das and Maite Taboada. 2018a. Rst signalling corpus: a corpus of signals of coherence relations. Language Resources and Evaluation, 52. ↗
  10. Debopam Das and Maite Taboada. 2018b. Signalling of coherence relations in discourse, beyond discourse markers. Discourse Processes, 55(8):743–770. ↗
  11. Debopam Das and Maite Taboada. 2019. Multiple signals of coherence relations. Discours. Revue de linguistique, psycholinguistique et informatique. A journal of linguistics, psycholinguistics and computational linguistics, (24). ↗
  12. Michelle L Gaddy, Paul van den Broek, and Yung-Chi Sung. 2001. The inxuence of text cues on the allocation of attention during reading. Text representation: Linguistic and psycholinguistic aspects, 8:89. find
  13. René Knaebel. 2021. Discopy: A neural system for shallow discourse parsing. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 128–133, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. ↗
  14. Yang Janet Liu, Tatsuya Aoyama, and Amir Zeldes. 2023. What’s hard in English RST parsing? predictive models for error analysis. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–42, Prague, Czechia. Association for Computational Linguistics.
  15. Erbo Shang, Xiaohua Liu, Hailong Wang, Yangfeng Rong, and Yuerong Liu. 2019. Research on the application of artificial intelligence and distributed parallel computing in archives classification. In 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), pages 1267–1271. find
  16. Manfred Stede and Arne Neumann. 2014. Potsdam commentary corpus 2.0: Annotation for discourse research. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 925–929, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  17. Amir Zeldes. 2017. The gum corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51:581–612. ↗
  18. Amir Zeldes, Yang Janet Liu, Mikel Iruskieta, Philippe Muller, Chloé Braud, and Sonia Badene. 2021. find
  19. The DISRPT 2021 shared task on elementary discourse unit segmentation, connective detection, and relation classification. In Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021), pages 1–12, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  20. Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. DMRST: A joint framework for document-level multilingual RST discourse segmentation and parsing. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 154–164, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. ↗
  21. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281. ↗
  22. Daniel Marcu. 2000. The rhetorical parsing of unrestricted texts: A surface-based approach. Computational Linguistics, 26(3):395–448. ↗
  23. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90, Manchester, UK. Coling 2008 Organizing Committee.
  24. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). ↗
  25. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014. Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pastor-2024-signals-as-features-predicting-error/