Elamur Research⌕
☰
home / papers / 2025 / Disagreements in analyses of rhetorical text structure: A new dataset and first analyses

Disagreements in analyses of rhetorical text structure: A new dataset and first analyses

Paper · 2025 · Freya Hewett and Manfred Stede

Summary

Rhetorical Structure Theory (RST) annotation is notoriously subjective, and low inter-annotator agreement is usually read as a quality problem. Hewett and Stede instead ask which disagreements are legitimate, meaning cases where two different trees are both defensible. They assemble 156 English and German texts, each analysed twice by trained annotators over the same EDU segmentation. The sources are RST-DT, UNSC-RST, APA-RST and PCC. They run RST-Tace to list every point where the two analyses diverge and sort these points into bins of annotation decisions. One author then judged a sample of 480 disagreements, decided whether each is an error or a valid alternative, and built a typology of causes.

The central conclusion is that two kinds of ambiguity should be kept apart:

  • Linguistic ambiguity, which is genuinely present in the text, for example vague coreference or the scope of attribution verbs.
  • Theory- or guideline-induced ambiguity, which comes from the framework itself, for example strong-nuclearity effects, overlapping relation definitions, or arbitrary choices such as stacking relations versus forming a List.

This matters for guideline design, for evaluating discourse parsers, and for perspectivist approaches to subjective tasks in general.

Key points

  • Dataset: 156 doubly-annotated texts from four sources.
    • RST-DT: Wall Street Journal articles.
    • UNSC-RST: UN Security Council speeches, 2014/15.
    • APA-RST: German news articles and their easy-language versions.
    • PCC: German newspaper commentaries.
    • APA and PCC follow the same guidelines and are merged as APA+PCC.
  • Tool: RST-Tace compares two trees with identical segmentation along four aspects: nuclearity, relation, constituent and attachment point. Unit pairs are matched through their central subconstituents.
  • Release: the parallel APA+PCC and UNSC data are available as rs3 XML plus a Tace-based csv in the RSTmulti GitHub repository. Because RST-DT is licensed through the LDC, only its text IDs are shared.
  • Qualitative analysis: 480 disagreements were inspected.
    • The status labels separate Vague from Both. Vague cases might be fixed by sharper definitions. Both cases are ones where two relations really apply at once.
    • Antithesis, Concession and Contrast are frequently confused with one another.
    • Elaboration overlaps with many other relations.
  • Limitations:
    • Segmentation is held fixed.
    • Only one expert served as judge.
    • Relation mismatches on identical spans were judged without the surrounding context.
    • Tace can only compare two annotations at a time.

Interesting ideas

  • I1. Break the comparison of two whole trees into a list of local disagreement points (nuclearity, relation, span, attachment), so that each divergence can be counted and inspected individually rather than summarised by one agreement score.
  • I2. Give every disagreement a status: an error, a vague case, or a case where both readings hold. This keeps genuine ambiguity from being treated as noise.
  • I3. Separate ambiguity that lives in the language from ambiguity created by the annotation scheme's own arbitrary choice points, and resolve the second kind by convention in the guidelines.
  • I4. Allow more than one relation on the same span instead of forcing a single label, for example a causal link together with Elaboration.
  • I5. Hold segmentation fixed so that the tree-building step can be analysed on its own before segmentation is studied.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when witness parsers disagree on silver data, record each conflicting head or label as a separate typed item, so that repair rules and evaluation can address the specific conflicts.
  • I2 (learn): when scoring against gold or teacher output, allow a "both acceptable" verdict so that plausible alternative attachments are not counted as failures during training or negative-control evaluation.
  • I3 (tree): in UD dialect conversion, record scheme-induced variants (such as list or coordination binarization) as deterministic conventions, distinct from true attachment ambiguities that should stay underspecified.
  • I4 (frames): let a clause pair carry several discourse or implicature relations at once, so that later reasoning can use whichever is most informative and the choice stays explainable.
  • I5 (world): when debugging event order or state tracking, keep the entity and event inventory frozen and compare only how events are linked, which isolates where two analyses of a story diverge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 worksLiterature and Literary Theory · 519 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDiscourse analysis · 466 works

Cite

Cite as Freya Hewett and Manfred Stede. 2025. Disagreements in analyses of rhetorical text structure: A new dataset and first analyses. Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.law-1.3

@inproceedings{hewett2025disagreements,
  title = {{Disagreements in analyses of rhetorical text structure: A new dataset and first analyses}},
  author = {Freya Hewett and Manfred Stede},
  booktitle = {Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025)},
  year = {2025},
  doi = {10.18653/v1/2025.law-1.3},
  url = {https://doi.org/10.18653/v1/2025.law-1.3}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics.
  2. Daniel Marcu. 2000. The Rhetorical Parsing of Unrestricted Texts: A Surface-based Approach. Computational Linguistics, 26(3):395–448. ↗
  3. Johanna D. Moore and Martha E. Pollack. 1992. A problem for RST: The need for multi-level discourse analysis. Computational Linguistics, 18(4):537–544.
  4. Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, and Amir Zeldes, editors. 2023. Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023). The Association for Computational Linguistics, Toronto, Canada. find
  5. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2018. A dependency perspective on RST discourse parsing and evaluation. Computational Linguistics, 44(2):197–235. ↗
  6. Martial Pastor and Nelleke Oostdijk. 2024. Signals as Features: Predicting Error/Success in Rhetorical Structure Parsing. In Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024), pages 139–148, St. Julians, Malta. Association for Computational Linguistics. ↗
  7. Lynn Carlson and Daniel Marcu. 2001. Discourse tagging reference manual. Technical report, Univ. of Southern California/ISI. Unpublished manuscript. find
  8. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Jan van Kuppevelt and Ronnie Smith, editors, Current Directions in Discourse and Dialogue. Kluwer, Dordrecht. ↗
  9. Sara Shahmohammadi and Manfred Stede. 2024. Discourse parsing for German with new RST corpora. In Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024), pages 65– 74, Vienna, Austria. Association for Computational Linguistics. find
  10. Freya Hewett. 2023. APA-RST: A text simplification corpus with RST annotations. In Proceedings of the 4th Workshop on Computational Approaches to Discourse (CODI 2023), pages 173–179, Toronto, Canada. Association for Computational Linguistics. find
  11. Manfred Stede. 2008. RST revisited: Disentangling nuclearity. In Cathrine Fabricius-Hansen and Wiebke Ramm, editors, ‘Subordination’ versus ‘coordination’ in sentence and text. John Benjamins, Amsterdam. find
  12. Patrick Huber, Wen Xiao, and Giuseppe Carenini. 2021. W-RST: Towards a weighted RST-style discourse framework. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3908–3918, Online. Association for Computational Linguistics. find
  13. Manfred Stede, Maite Taboada, and Debopam Das. 2017. Annotation Guidelines for Rhetorical Structure. Unpublished manuscript. find
  14. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385– 1470. ↗
  15. Mikel Iruskieta, Iria da Cunha, and Maite Taboada. 2015. A qualitative comparison method for rhetorical structures: identifying different discourse structures in multilingual corpora. Language Resources and Evaluation, 49(2):263–309. ↗
  16. Shujun Wan, Tino Kutschbach, Anke Lüdeling, and Manfred Stede. 2019. RST-Tace A tool for automatic comparison and evaluation of RST trees. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 88–96, Minneapolis, MN. Association for Computational Linguistics. find
  17. Yang Janet Liu, Tatsuya Aoyama, and Amir Zeldes. 2023. What’s Hard in English RST Parsing? Predictive Models for Error Analysis. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–42, Prague, Czechia. Association for Computational Linguistics.
  18. Karolina Zaczynska and Manfred Stede. 2024. Rhetorical strategies in the UN security council: Rhetorical Structure Theory and conflicts. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 15–28, Kyoto, Japan. Association for Computational Linguistics. find
  19. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. ↗
  20. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  21. Marian Marchal, Merel Scholman, Frances Yung, and Vera Demberg. 2022. Establishing annotation quality in multi-label annotations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3659–3668, Gyeongju, Republic of Amir Zeldes, Tatsuya Aoyama, Yang Janet Liu, Siyao Peng, Debopam Das, and Luke Gessler. 2024. eRST: A Signaled Graph Theory of Discourse Relations and Organization. Computational Linguistics, pages 1–47. ↗
  22. Šárka Zikánová. 2024. Text Structure and Its Ambiguities: Corpus Annotation as a Helpful Guide. In Proceedings of the 24th Conference Information Technologies – Applications and Theory (ITAT 2024), pages 2–12, Drienica, Slovakia. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hewett-2025-disagreements-analyses-rhetorical-text-structure/