Elamur Research
☰
home / papers / 2001 / Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory

Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory

What it is about

The paper describes how the RST Discourse Treebank was built: 385 American English articles from the Penn Treebank, each annotated with a rhetorical structure tree in Rhetorical Structure Theory. The authors explain why RST was chosen (it gives rich annotation of intention, semantics and text, and earlier work showed high agreement). They report the training, the phases of the project and the quality control. Agreement is measured with kappa over hierarchical structures, following Marcu et al. (1999). Averaged over three annotators, the kappa values are reported in tables for each stage and in a final table for six taggers. The authors use Krippendorff's threshold: kappa above 0.8 is very high agreement, 0.6 to 0.8 is good. The corpus took more than a dozen people about one year to build, and an annotation of one document took 30 minutes to several hours.

Ideas

For Mova

The chapter is languages/en/36-discourse-relations.md. The code is tools/treeagree, and a future discourse module in prag. The measure is kappa per phase on a shared set of documents. The RST Discourse Treebank is distributed by the LDC under its own licence, so it is for evaluation only, and no text is copied into a release.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W01-1605.pdf

Summary

Large annotated corpora had already changed NLP research, but rich theories of discourse structure had mostly been tested on small samples or used for shallow, segment-level labelling. The authors set out to build a large reference corpus with consistent discourse annotation. They chose Rhetorical Structure Theory (RST) for three reasons: it captures intentional, semantic and textual relations; earlier work showed that annotators could agree reasonably well on it; and RST trees had already helped in generation, summarization and machine translation. Their protocol uses clauses as elementary discourse units (EDUs). Explicit rules cover complement clauses, embedded units and a small number of phrasal units that start with a strong discourse marker. Adjacent spans are then joined into a tree, and each node is labelled with nuclearity and a rhetorical relation. Professional language analysts went through three phases of training, during which the guidelines were refined and agreement was tracked. The resulting RST Corpus is built on Wall Street Journal texts from the Penn Treebank and is distributed through the Linguistic Data Consortium. Agreement was highest for segmentation and fell at higher levels of the tree. The corpus adds a discourse layer on top of the existing syntactic annotation.

Key points

  • The RST Corpus contains 385 WSJ articles from the Penn Treebank: over 176,000 words and 21,789 EDUs. On average a document has 458 words and about 56.6 EDUs, and an EDU has about 8.1 words.
  • To measure inter-annotator consistency, 53 documents (13.8%) were annotated twice. Agreement dropped as decisions moved from the clause level to links between sentences and then between large spans.
  • The annotation uses 78 relations (53 mononuclear, 25 multinuclear), grouped into 16 classes such as Attribution, Cause, Elaboration and Topic-Comment. Three more relations give the tree its structure: textual-organization, span and same-unit.
  • The work took more than a dozen people from January to December 2000. A single document took anywhere from about 30 minutes to several hours.
  • The corpus, tagging tool and guidelines are available through LDC and isi.edu~marcu/discourse. The authors point to limits of RST: it uses one method for every level of the tree, it forces a tree structure, and nuclearity leaves room for interpretation.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1118078.1118083

@inproceedings{carlson2001building,
  title = {{Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory}},
  author = {Carlson, Lynn and Marcu, Daniel and Okurovsky, Mary Ellen},
  booktitle = {Proceedings of the Second SIGdial Workshop on Discourse and Dialogue},
  year = {2001},
  doi = {10.3115/1118078.1118083},
  url = {https://doi.org/10.3115/1118078.1118083}
}

References

39 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bruce Britton and John Black. 1985. Understanding Expository Text. Hillsdale, NJ: Lawrence Erlbaum Associates. find
  2. Jill Burstein, Daniel Marcu, Slava Andreyev, and Martin Chodorow. 2001. Towards automatic identification of discourse elements in essays. In Proceedings of the 39 th Annual Meeting of the Association for Computational Linguistics, Toulouse, France. find
  3. Jean Carletta, Amy Isard, Stephen Isard, Jacqueline Kowtko, Gwyneth Doherty-Sneddon, and Anne Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational Linguistics 23(1): 13-32. find
  4. Giacomo Ferrari. 1998. Preliminary steps toward the creation of a discourse and text resource. In Proceedings of the First International Conference on Language Resources and Evaluation (LREC 1998), Granada, Spain, 999-1001. find
  5. Giovanni Flammia and Victor Zue. 1995. Empirical evaluation of human performance and agreement in parsing discourse constituents in spoken dialogue. In Proceedings of the 4 th European Conference on Speech Communication and Technology, Madrid, Spain, vol. 3, 1965-1968. find
  6. Roger Garside, Steve Fligelstone and Simon Botley. 1997. Discourse Annotation: Anaphoric Relations in Corpora. In Corpus annotation: Linguistic information from computer text corpora, edited by R. Garside, G. Leech, and T. McEnery. London: Longman, 66-84. find
  7. Talmy Givon. 1983. Topic continuity in discourse. In Topic Continuity in Discourse: a Quantitative Cross-Language Study. Amsterdam/Philadelphia: John Benjamins, 1-41. find
  8. Joseph Evans Grimes. 1975. The Thread of Discourse. The Hague, Paris: Mouton. find
  9. Barbara Grosz and Candice Sidner. 1986. Attentions, intentions, and the structure of discourse. Computational Linguistics, 12(3): 175-204. find
  10. Marti Hearst. 1997. TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics 23(1): 33-64. find
  11. Julia Hirschberg and Diane Litman. 1993. Empirical studies on the disambiguation of cue phrases. Computational Linguistics 19(3): 501-530. find
  12. Eduard Hovy. 1993. Automated discourse generation using discourse structure relations. Artificial Intelligence 63(1-2): 341-386. find
  13. Klaus Krippendorff. 1980. Content Analysis: An Introduction to its Methodology. Beverly Hills, CA: Sage Publications. find
  14. Geoffrey Leech, Tony McEnery, and Martin Wynne. 1997. Further levels of annotation. In Corpus Annotation: Linguistic Information from Computer Text Corpora, edited by R. Garside, G. Leech, and T. McEnery. London: Longman, 85-101. find
  15. Robert Longacre. 1983. The Grammar of Discourse. New York: Plenum Press. find
  16. William Mann and Sandra Thompson. 1988. Rhetorical structure theory. Toward a functional theory of text organization. Text, 8(3): 243-281. find our card
  17. William Mann and Sandra Thompson, eds. 1992. Discourse Description: Diverse Linguistic Analyses of a Fund-raising Text. Amsterdam/Philadelphia: John Benjamins. find
  18. Daniel Marcu. 2000. The Theory and Practice of Discourse Parsing and Summarization. Cambridge, MA: The MIT Press. find
  19. Daniel Marcu, Estibaliz Amorrortu, and Magdelena Romera. 1999. Experiments in constructing a corpus of discourse trees. In Proceedings of the ACL Workshop on Standards and Tools for Discourse Tagging, College Park, MD, 48-57. find
  20. Daniel Marcu, Lynn Carlson, and Maki Watanabe. 2000. The automatic translation of discourse structures. Proceedings of the First Annual Meeting of the North American Chapter of the Association for Computational Linguistics, Seattle, WA, 9-17. find
  21. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank, Computational Linguistics 19(2), 313-330. find our card
  22. Bonnie Meyer. 1985. Prose Analysis: Purposes, Procedures, and Problems. In Understanding Expository Text, edited by B. Britton and J. Black. Hillsdale, NJ: Lawrence Erlbaum Associates, 11-64. find
  23. Johanna Moore. 1995. Participating in Explanatory Dialogues: Interpreting and Responding to Questions in Context. Cambridge, MA: MIT Press. find
  24. Johanna Moore and Cecile Paris. 1993. Planning text for advisory dialogues: capturing intentional and rhetorical information. Computational Linguistics 19(4): 651-694. find
  25. Megan Moser and Johanna Moore. 1995. Investigating cue selection and placement in tutorial discourse. Proceedings of the 33 rd Annual Meeting of the Association for Computational Linguistics, Cambridge, MA, 130-135. find
  26. Tadashi Nomoto and Yuji Matsumoto. 1999. Learning discourse relations with active data selection. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, College Park, MD, 158-167. find
  27. Rebecca Passonneau and Diane Litman. 1997. Discourse segmentation by human and automatic means. Computational Linguistics 23(1): 103-140. find
  28. Marie-Paule Pery-Woodley and Josette Rebeyrolle. 1998. Domain and genre in sublanguage text: definitional microtexts in three corpora. In Proceedings of the First International Conference on Language Resources and Evaluation (LREC-1998), Granada, Spain, 987-992. find
  29. Livia Polanyi. 1988. A formal model of the structure of discourse. Journal of Pragmatics 12: 601-638. find
  30. Livia Polanyi. 1996. The linguistic structure of discourse. Center for the Study of Language and Information. CSLI-96-200. find
  31. Josette Rebeyrolle. 2000. Utilisation de contextes définitoires pour l’acquisition de connaissances à partir de textes. In Actes Journées Francophones d’Ingénierie de la Connaissance (IC’2000), Toulouse, IRIT, 105-114. find
  32. Harvey Sacks, Emmanuel Schegloff, and Gail Jefferson. 1974. A simple systematics for the organization of turntaking in conversation. Language 50: 696-735. find
  33. Sidney Siegal and N.J. Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill. find
  34. Beth Sundheim. 1995. Overview of results of the MUC-6 evaluation. In Proceedings of the Sixth Message Understanding Conference (MUC-6), Columbia, MD, 13-31. find
  35. Benjamin K. T’sou, Tom B.Y. Lai, Samuel W.K. Chan, Weijun Gao, and Xuegang Zhan. 2000. Enhancement of Chinese discourse marker tagger with C.4.5. In Proceedings of the Second Chinese Language Processing Workshop, Hong Kong, 38-45. find
  36. Teun A. Van Dijk and Walter Kintsch. 1983. Strategies of Discourse Comprehension. New York: Academic Press. find
  37. Ellen Voorhees and Donna Harman. 1999. The Eighth Text Retrieval Conference (TREC-8). NIST Special Publication 500-246. find
  38. Charles Wayne. 2000. Multilingual topic detection and tracking: successful research enabled by corpora and evaluation. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC-2000), Athens, Greece, 1487-1493. find
  39. Janyce Wiebe, Rebecca Bruce, and Thomas O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proceedings of the 37 th Annual Meeting of the Association for Computational Linguistics. College Park, MD, 246-253. find