Elamur Research⌕
☰
home / papers / 2019 / How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations

How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations

Paper · 2019 · Vera Demberg, Merel C.J. Scholman and Fatemeh Torabi Asr

Summary

Discourse corpora are annotated under incompatible frameworks, so researchers cannot easily search them together or pool them to train classifiers. Several mappings between relation labels have been proposed on theoretical grounds, but none had been checked against real annotations. This paper does that check using the overlap between the Penn Discourse Treebank (PDTB 2.0) and the RST Discourse Treebank (RST-DT), which annotate some of the same texts. The authors build an algorithm that aligns discourse segments across the two treebanks. They then compare the label pairs that actually co-occur with the correspondences predicted by three proposals: Chiarcos (2014), Bunt & Prasad (2016), and Sanders et al. (2018).

Segmentation strongly shapes which relation labels end up being assigned. The frameworks agree reasonably well on explicit relations but poorly on implicit ones. The authors trace systematic discrepancies to differences in operationalization and annotation goals: RST-DT encodes author intention, while PDTB encodes logical-semantic links. These results matter for future annotation guidelines, for cross-corpus search, and for training discourse parsers. The aligned corpus is released.

Key points

  • The study uses the shared texts of PDTB 2.0 (Prasad et al., 2008) and RST-DT (Carlson et al., 2003). It evaluates three mapping schemes and relates its findings to an earlier, smaller study by Rehbein et al. (2016).
  • Recurring mismatches include:
    • certain RST-DT causal labels appearing as additive relations in PDTB;
    • RST's CONTRAST frequently corresponding to PDTB's CONCESSION.
  • For ambiguous connectives, the two corpora agree when a connective can signal very different relations (e.g., while: causal vs. temporal). They disagree more when a connective signals closely related relations (e.g., but: contrast vs. concession).
  • The authors use the terms ideational vs. intentional for the two levels on which a relation can hold. A single segment can serve both functions at once, such as being an example and also evidence for a claim.
  • As context, implicit relation classification in the CoNLL shared tasks reached only a little over 40% F-score in an 11-way setup. The authors argue this should be interpreted in light of how inconsistent the human labels themselves are.

Interesting ideas

  • I1. Instead of trusting a label correspondence derived from definitions, run it over text annotated under both schemes and inspect which label pairs actually co-occur. This exposes gaps between the guidelines and annotation practice.
  • I2. Align segments before comparing labels, and report boundary mismatches as a separate error source, because segmentation alone can produce apparent label disagreement.
  • I3. Test whether two annotations agree more than the distribution of the surface marker alone would predict. This chance baseline distinguishes genuine agreement from agreement inherited from the cue word.
  • I4. Keep separate ideational (content) and intentional (speaker-goal) relation layers, and allow both to hold for the same pair of segments.
  • I5. When scoring, accept a set of equally valid labels, or weight errors by how often humans substitute one label for another. Cross-framework agreement can also sort instances into easy and hard.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Apply this to UD dialect conversion: parse sentences present in two treebanks, convert one side, and tabulate where relation labels diverge, rather than trusting the hand-written conversion rules.
  • I2 (learn): When multiple witness parsers vote on silver data, first align their tokenization and spans, so that tokenization disagreements are not counted as relation disagreements.
  • I3 (learn): Use the per-connective label distribution as a negative-control baseline when judging whether a relation classifier has learned anything beyond the connective.
  • I4 (frames): In prag, a clause could carry both a content relation (cause, example) and a speech-act purpose (evidence, justification) without one overriding the other.
  • I5 (answer): Grade relation and reason questions on fables using a set of acceptable answers with graded penalties, and report scores separately on easy and contested items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany102 works · 17,447 citations🎓Simon Fraser UniversityBurnaby · Canada10 works · 4,261 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

🧭 Topics 1 Discourse analysis · 394 works

Cite

Cite as Vera Demberg, Merel C.J. Scholman and Fatemeh Torabi Asr. 2019. How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations. Dialogue & Discourse.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5087/dad.2019.104

@inproceedings{demberg2019how,
  title = {{How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations}},
  author = {Vera Demberg, Merel C.J. Scholman and Fatemeh Torabi Asr},
  booktitle = {Dialogue \& Discourse},
  year = {2019},
  doi = {10.5087/dad.2019.104},
  url = {https://doi.org/10.5087/dad.2019.104}
}

References

56 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Saif, A., & Markert, K. (2010). The Leeds Arabic Discourse Treebank: Annotating Discourse Connectives for Arabic. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC) (pp. 2046–2053). Valletta, Malta. our card
  2. Asr, F. T., & Demberg, V. (2013). On the information conveyed by discourse markers. In Proceedings of the Fourth Annual Workshop on Cognitive Modeling and Computational Linguistics (pp. 84–93). our card
  3. Benamara, F., & Taboada, M. (2015). Mapping different rhetorical relation annotations: A proposal. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics,* SEM (pp. 147–152). find
  4. Blakemore, D. (1997). Restatement and exemplification: A relevance theoretic reassessment of elaboration. Pragmatics & Cognition, 5, 1–19. find
  5. Bunt, H., & Prasad, R. (2016). ISO-DR-Core (ISO 24617-8): Core concepts for the annotation of discourse relations. In Proceedings 12th Joint ACL-ISO Workshop on Interoperable Semantic Annotation (ISA-12) (pp. 45–54). find
  6. Cardoso, P. C., Maziero, E. G., Jorge, M. L., Seno, E. M., Di Felippo, A., Rino, L. H., Nunes, M. G., & Pardo, T. A. (2011). CSTnews – a discourse-annotated corpus for single and multi-document summarization of news texts in Brazilian Portuguese. In Proceedings of the 3rd RST Brazilian Meeting (pp. 88–105). find
  7. Carlson, L., & Marcu, D. (2001). Discourse tagging reference manual. find
  8. Carlson, L., Marcu, D., & Okurowski, M. E. (2003). Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Current and new directions in discourse and dialogue (pp. 85–112). Springer. ↗ our card
  9. Carston, R. (1993). Conjunction, explanation and relevance. Lingua, 90, 27–48. ↗
  10. Chiarcos, C. (2014). Towards interoperable discourse annotation. Discourse features in the Ontologies of Linguistic Annotation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014) (pp. 4569–4577). find
  11. Crible, L., & Degand, L. (2017). Reliability vs. granularity in discourse annotation: What is the trade-off? Corpus Linguistics and Linguistic Theory, . ↗
  12. Das, D., & Taboada, M. (2018). Rst signalling corpus: A corpus of signals of coherence relations. Language Resources and Evaluation, 52, 149–184. ↗
  13. Hoek, J., Evers-Vermeul, J., & Sanders, T. J. (2017). Segmenting discourse: Incorporating interpretation into segmentation? Corpus Linguistics and Linguistic Theory, Advance online publication. ↗
  14. Hovy, E. H., & Maier, E. (1995). Parsimonious or profligate: How many and which discourse structure relations. Unpublished manuscript, . find
  15. Iruskieta, M., Aranzabe, M. J., de Ilarraza, A. D., Gonzalez, I., Lersundi, M., & de Lacalle, O. L. (2013). The RST Basque TreeBank: an online search interface to check rhetorical relations. In 4th workshop RST and Discourse Studies (pp. 40–49). find
  16. Jansen, P., Surdeanu, M., & Clark, P. (2014). Discourse complements lexical semantics for non-factoid answer reranking. In ACL (1) (pp. 977–986). find
  17. Lee, A., Prasad, R., Joshi, A., Dinesh, N., & Webber, B. (2006). Complexity of dependencies in discourse: Are dependencies in discourse more complex than in syntax? In Proceedings of the 5th International Workshop on Treebanks and Linguistic Theories (TLT), Prague, Czech Republic (pp. 79–90). find
  18. Lee, A., Prasad, R., Joshi, A., & Webber, B. (2008). Departures from tree structures in discourse: Shared arguments in the penn discourse treebank. In Proceedings of the Constraints in Discourse III Workshop (pp. 61–68). our card
  19. Liu, Y., Li, S., Zhang, X., & Sui, Z. (2016). Implicit discourse relation classification via multi-task neural networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) (pp. 2750–2756). our card
  20. Mann, W. C., & Thompson, S. A. (1988). Rhetorical Structure Theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8, 243–281. ↗ our card
  21. Marcu, D. (2000). The theory and practice of discourse parsing and summarization. MIT press. ↗
  22. Meyer, T., & Popescu-Belis, A. (2012). Using sense-labeled discourse connectives for statistical machine translation. In Proceedings of the Joint Workshop on Exploiting Synergies between Information Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches to Machine Translation (HyTra) (pp. 129–138). Association for Computational Linguistics. our card
  23. Moore, J. D., & Pollack, M. E. (1992). A problem for RST: The need for multi-level discourse analysis. Computational Linguistics, 18, 537–544. our card
  24. Oza, U., Prasad, R., Kolachina, S., Sharma, D. M., & Joshi, A. (2009). The Hindi discourse relation bank. In Proceedings of the third Linguistic Annotation Workshop (LAW) (pp. 158–161). Association for Computational Linguistics. our card
  25. Pitler, E., Raghupathy, M., Mehta, H., Nenkova, A., Lee, A., & Joshi, A. K. (2008). Easily identifiable discourse relations. Technical report. our card
  26. Polakova, L., Mirovsky, J., & Synkova, P. (2017). Signalling implicit relations: A pdtb-rst comparison. Dialogue & Discourse, 8, 225–248. ↗
  27. Popescu-Belis, A. (2016). Manual and automatic labeling of discourse connectives for machine translation. In TextLink–Structuring Discourse in Multilingual Europe Second Action Conference Károli Gáspár University of the Reformed Church in Hungary Budapest, 11–14 April, 2016 (p. 16). find
  28. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A. K., & Webber, B. (2008). The Penn Discourse TreeBank 2.0. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). Citeseer. our card
  29. Prasad, R., Forbes-Riley, K., & Lee, A. (2017). Towards full text shallow discourse relation annotation: Experiments with cross-paragraph implicit relations in the PDTB. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue (pp. 7–16). our card
  30. Prasad, R., Miltsakaki, E., Dinesh, N., Lee, A., Joshi, A. K., Robaldo, L., & Webber, B. (2007). The Penn Discourse Treebank 2.0 annotation manual. our card
  31. Prasad, R., Webber, B., & Joshi, A. (2014a). Reflections on the Penn Discourse Treebank, comparable corpora, and complementary annotation. Computational Linguistics, . ↗ our card
  32. Prasad, R., Webber, B., & Joshi, A. (2014b). Reflections on the Penn Discourse TreeBank, Comparable Corpora, and Complementary Annotation. Computational Linguistics, 40, 921–950. ↗ our card
  33. Prasad, R., Webber, B., & Lee, A. (2018). Discourse annotation in the PDTB: The next generation. In Proceedings 14th Joint ACL-ISO Workshop on Interoperable Semantic Annotation (pp. 87–97). our card
  34. Redeker, G. (1990). Ideational and pragmatic markers of discourse structure. Journal of Pragmatics, 14, 367–381. ↗
  35. Rehbein, I., Scholman, M. C. J., & Demberg, V. (2016). Annotating discourse relations in spoken language: A comparison of the PDTB and CCR frameworks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC). Portoroz, Slovenia: European Language Resources Association (ELRA). our card
  36. Robaldo, L., & Miltsakaki, E. (2014). Corpus-driven semantics of concession: Where do expectations come from? Dialogue & Discourse, 5, 1–36. ↗
  37. Sanders, T. J. M., Demberg, V., Hoek, J., Scholman, M. C. J., Torabi Asr, F., Zufferey, S., & Evers-Vermeul, J. (2018). Unifying dimensions in discourse relations: How various annotation frameworks are related. Corpus Linguistics and Linguistic Theory, ahead of print, 1–71. ↗
  38. Sanders, T. J. M., & Spooren, W. (1999). Communicative intentions and coherence relations. In W. Bublits, U. Lenk, & E. Ventola (Eds.), Coherence in Spoken and Written Discourse (pp. 235– 250). John Benjamins. ↗
  39. Sanders, T. J. M., Spooren, W. P. M. S., & Noordman, L. G. M. (1992). Toward a taxonomy of coherence relations. Discourse Processes, 15, 1–35. ↗
  40. Scheffler, T., & Stede, M. (2016a). Adding semantic relations to a large-coverage connective lexicon of german. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). find
  41. Scheffler, T., & Stede, M. (2016b). Mapping PDTB-style connective annotation to RST-style discourse annotation. In Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016). find
  42. Scholman, M. C. J., & Demberg, V. (2017). Examples and specifications that prove a point: Identifying elaborative and argumentative discourse relations. Dialogue & Discourse, 8, 56–83. ↗ our card
  43. Sharp, R., Jansen, P., Surdeanu, M., & Clark, P. (2015). Spinning straw into gold: Using free text to train monolingual alignment models for non-factoid question answering. In HLT-NAACL (pp. 231–237). find
  44. Somasundaran, S., Namata, G., Wiebe, J., & Getoor, L. (2009). Supervised and unsupervised methods in employing discourse relations for improving opinion polarity classification. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1 (pp. 170–179). Association for Computational Linguistics. our card
  45. Stede, M. (2008). RST revisited: Disentangling nuclearity. Subordinationversus Coordinationin Sentence and Text, (pp. 33–59). ↗
  46. Stede, M., & Neumann, A. (2014a). Potsdam Commentary Corpus 2.0: Annotation for discourse research. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014) (pp. 925–929). find
  47. Stede, M., & Neumann, A. (2014b). Potsdam Commentary Corpus 2.0: Annotation for discourse research. In LREC (pp. 925–929). find
  48. Taboada, M., & Mann, W. C. (2006). Rhetorical Structure Theory: Looking back and moving ahead. Discourse Studies, 8, 423–459. ↗
  49. Tonelli, S., Riccardi, G., Prasad, R., & Joshi, A. K. (2010). Annotation of discourse relations for conversational spoken dialogs. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC) (pp. 2084–2090). Marrakesh, Marocco. our card
  50. Webber, B., Prasad, R., Lee, A., & Joshi, A. (2016). A discourse-annotated corpus of conjoined VPs. In Proceedings of the 10th Linguistic Annotation Workshop (LAW-X) (pp. 22–31). find
  51. Xue, N., Ng, H. T., Pradhan, S., Prasad, R., Bryant, C., & Rutherford, A. (2015). The CoNLL-2015 shared task on shallow discourse parsing. In CoNLL Shared Task (pp. 1–16). our card
  52. Xue, N., Ng, H. T., Rutherford, A., Webber, B., Wang, C., & Wang, H. (2016). CoNLL 2016 shared task on multilingual shallow discourse parsing. Proceedings of the CoNLL-16 shared task, (pp. 1–19). our card
  53. Zhou, L., Li, B., Gao, W., Wei, Z., & Wong, K.-F. (2011). Unsupervised discovery of discourse relations for eliminating intra-sentence polarity ambiguities. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 162–171). Association for Computational Linguistics. our card
  54. Zhou, Y., & Xue, N. (2015). The Chinese Discourse TreeBank: A Chinese corpus annotated with discourse relations. Language Resources and Evaluation, 49, 397–431. ↗
  55. Zirn, C., Niepert, M., Stuckenschmidt, H., & Strube, M. (2011). Fine-grained sentiment analysis with structural features. In IJCNLP (pp. 336–344). find
  56. Zufferey, S., & Degand, L. (2013). Annotating the meaning of discourse connectives in multilingual corpora. Corpus Linguistics and Linguistic Theory, 1, 1–24. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/demberg-2019-how-compatible-are-our-discourse/