Elamur Research⌕
☰
home / papers / 2017 / The Good, the Bad, and the Disagreement: Complex ground truth in rhetorical structure analysis

The Good, the Bad, and the Disagreement: Complex ground truth in rhetorical structure analysis

Summary

The paper challenges the idea that RST annotation has one correct tree per text. Because rhetorical analysis involves subjective judgement, the authors argue that some annotator disagreement is legitimate and belongs in the gold standard. Such disagreement must be separated from errors made by annotators who misapplied the guidelines.

They ran two agreement studies using guidelines close to original RST. The German study used texts from the Potsdam Commentary Corpus. The English study used seven texts from the RST Discourse Treebank (e.g., wsj_1117). Pre-segmented EDUs were given to annotators with different amounts of training. Agreement was scored with Marcu's (2000) span/nuclearity/relation method, excluding single-EDU spans. Expert annotators agreed far more than lightly trained ones, especially on nuclearity and relation labels.

The authors then encoded competing expert trees in the Underspecified Rhetorical Markup Language (URML). They show it can hold all observed legitimate alternatives in one "complex gold" structure. This matters for building and evaluating RST parsers against ground truth that admits ambiguity.

Key points

  • The relation set has 31 relations, compared with 78 in RST-DT. It omits relations such as TOPIC-COMMENT and ATTRIBUTION and avoids SAME-UNIT. Segmentation followed the SLSeg guidelines.

  • The German study used 10 commentary texts and four annotators: two extensively trained (GE1, GE2) and two lightly trained (GL1, GL2). Percent agreement was:

    GroupSpanNuclearityRelation
    Experts (GE1–GE2)65.643.724.0
    Lightly trained (GL1–GL2)51.625.49.7

  • Taking GE1 as gold, GL1 scored precision = recall of 0.65 (spans), 0.56 (nuclearity) and 0.30 (relations).

  • URML supports several ways of recording alternatives:

    • shared subtrees via common node IDs;
    • ambiguity packing via node groups;
    • optional relation names and nuclearity;
    • per-alternative scores.

    The authors used structure sharing and packing to merge the trees of English experts EE1 and EE2.

  • Planned future work:

    • tools to convert and merge RSTTool rs3 files into URML;
    • agreement metrics that compare a new annotator or an RST parser against the complex gold.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-3602

@inproceedings{das2017good,
  title = {{The Good, the Bad, and the Disagreement: Complex ground truth in rhetorical structure analysis}},
  author = {Das, Debopam and Stede, Manfred and Taboada, Maite},
  booktitle = {Proceedings of the 6th Workshop on Recent Advances in RST and Related Formalisms},
  year = {2017},
  doi = {10.18653/v1/w17-3602},
  url = {https://doi.org/10.18653/v1/w17-3602}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proc. of the 6th International Conference on Language Resources and Evaluation (LREC). Marrakech, Morocco. find our card
  2. Nicholas Asher and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press, Cambridge. find
  3. Chloé Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Proceedings of the 26th International Conference on Computational Linguistics (COLING). Osaka, Japan, pages 1903–1913. find
  4. David Reitter. 2003. Simple signals for complex rhetorics: On rhetorical analysis with rich-feature support vector models. LDV Forum 18(1/2):38–52. find
  5. David Reitter and Manfred Stede. 2003. Step by step: underspecified markup in incremental rhetorical analysis. In Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora (LINC). Budapest. find
  6. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Jan van Kuppevelt and Ronnie Smith, editors, Current Directions in Discourse and Dialogue, Kluwer, Dordrecht, pages 85–112. find our card
  7. Frank Schilder. 2002. Robust discourse parsing via discourse markers, topicality and position. Natural Language Engineering 8(2-3):235–255. find
  8. Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, MA, pages 511–521. find our card
  9. Manfred Stede. 2016. Rhetorische Struktur. In Manfred Stede, editor, Handbuch Textannotation: Pots-damer Kommentarkorpus 2.0, Universitätsverlag, Potsdam. find
  10. Iskra Fodor. 2015. Übereinstimmung zwischen Annota-toren als Ähnlichkeitsmaß für RST-Bäume-Eine empirische Studie mit Texten aus dem Potsdam Commentary Corpus. B.Sc. Thesis, University of Potsdam. find
  11. Maite Taboada and William Mann. 2006. Rhetorical Structure Theory: Looking back and moving ahead. Discourse Studies 8(4):423–459. find
  12. Milan Tofiloski, Julian Brooke, and Maite Taboada. 2009. A syntactic and lexical-based discourse segmenter. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (Short Papers). Suntec, Sing… find
  13. Hugo Hernault, Hemut Prendinger, David duVerle, and Mitsuru Ishizuka. 2010. Hilda: A discourse parser using support vector machine classification. Dialogue and Discourse 1(3):1–33. find our card
  14. Mikel Iruskieta, Iria da Cunha, and Maite Taboada. 2015. Principles of a qualitative method for rhetorical analysis evaluation: A contrastive analysis english-spanish-basque. Language Resources and Evaluation 49(2):263–309. find
  15. Nynke van der Vliet, Ildikó Berzlánovich, Gosse Bouma, Markus Egg, and Gisela Redeker. 2011. Building a discourse-annotated Dutch text corpus. In Proceedings of the Workshop Beyond Semantics: Corpus-based annotations of Pragmatics and Discourse Phenomena. Göttingen, Germany, pages 157–171. find
  16. Yangfeng Ji and Jacob Eisenstein. 2013. Discriminative improvements to distributional sentence similarity. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 891–896. find
  17. William Mann and Sandra Thompson. 1988. Rhetorical Structure Theory: Towards a functional theory of text organization. Text 8:243–281. find
  18. Daniel Marcu. 2000. The theory and practice of discourse parsing and summarization. MIT Press, Cambridge/MA. find
  19. Johanna Moore and Martha Pollack. 1992. A problem for RST: The need for multi-level discourse analysis. Computational Linguistics 18(4):537–544. find
  20. Michael O’Donnell. 2000. RSTTool 2.4 – A markup tool for Rhetorical Structure Theory. In Proc. of the International Natural Language Generation Conference. Mizpe Ramon/Israel, pages 253–256. find our card

↑ back to top

This page in Markdown →