Elamur Research⌕
☰
home / papers / 2021 / Is there less annotator agreement when the discourse relation is underspecified?

Is there less annotator agreement when the discourse relation is underspecified?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.discann-1.1.pdf

Summary

People who annotate coherence relations tend to agree less on implicit relations than on relations marked by a connective. The authors test one explanation: implicit relations require more inference. If that were the main cause, agreement should drop as the marker becomes less specific.

They sampled 350 relations from English TED talk transcripts, drawn from TED-MDB and the Ted Corpus Search Engine:

  • 150 marked by specific connectives (because, in addition, even though)
  • 100 marked by underspecified ones (and, but)
  • 100 with no connective

Two expert coders labelled every relation using the CCR framework, and a third coder settled disagreements. Agreement on full labels was measured with percentage agreement, Cohen's Kappa and Gwet's AC1.

Explicit relations did reach higher agreement than implicit ones. However, underspecified relations did not land in between. Relations with and scored about as well as those with specific connectives, while but produced the weakest agreement of all. Connective specificity alone therefore cannot explain why discourse annotation is hard, which matters for guideline design and for interpreting corpus labels.

Key points

  • Data sources: All 47 but-relations and 50 and-relations came from TED-MDB, plus 3 extra but-relations from the search engine. The annotated dataset was released publicly.

  • Agreement scores (%, κ, AC1):

    Marker%κAC1
    because84.68.68
    and74.58.71
    implicit66.58.64
    but58.39.46

  • Why and beat but: The authors link this to and's stronger default reading. 78% of and-relations were positive additive, compared with 66% of but-relations being negative additive.

  • Source of but disagreements: Most concerned BASIC OPERATION, i.e. the notoriously hard split between contrast (negative additive) and concession (negative causal).

  • Implicit relations: These showed the widest variety of labels, and several relations can hold at once. ORDER OF THE SEGMENTS was excluded from the analysis because it was trivial for specific connectives (100% agreement).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hoek2021is,
  title = {{Is there less annotator agreement when the discourse relation is underspecified?}},
  author = {Hoek, Jet and Scholman, Merel C.J. and Sanders, Ted J.M.},
  booktitle = {Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation},
  year = {2021},
  url = {https://aclanthology.org/2021.discann-1.1/}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Fatemeh Torabi Asr and Vera Demberg. 2012. Implicitness of discourse relations. In Proceedings of COL-ING 2012, pages 2669–2684. find our card
  3. Bruce K. Britton. 1994. Understanding expository text: Building mental structures to induce insights. Academic Press. find
  4. Anneloes R. Canestrelli, Willem M. Mak, and Ted J.M. Sanders. 2013. Causal connectives in discourse processing: How differences in subjectivity are reflected in eye movements. Language and Cognitive Processes, 28(9):1394–1413. find
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  6. Ludivine Crible, Ágnes Abuczki, Nijolė Burkšaitienė, Péter Furkó, Anna Nedoluzhko, Sigita Rack-evičienė, Giedrė Valūnaitė Oleškevičienė, and Šárka Zikánová. 2019. Functions and translations of discourse markers in ted talks: A parallel corpus study of underspecification in five lan… find
  7. Liesbeth Degand and Sandrine Zufferey. 2013. Annotating the meaning of discourse connectives in multilingual corpora. Corpus linguistics and linguistic theory, pages 1–24. find
  8. Jacqueline Evers-Vermeul, Jet Hoek, and Merel C.J. Scholman. 2017. On temporality in discourse annotation: Theoretical and practical considerations. Dialogue & Discourse, 8(2):1–20. find
  9. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of clinical epidemiology, 43(6):543–549. find
  10. Morton Ann Gernsbacher. 1997. Coherence cues mapping during comprehension. Processing inter-clausal relationships. Studies in the production and comprehension of text, pages 3–22. find
  11. Kilem Gwet. 2001. Handbook of inter-rater reliability: How to estimate the level of agreement between two or multiple raters. Gaithersburg, MD: STATAXIS Publishing Company. find
  12. Yoichiro Hasebe. 2015. Design and implementation of an online corpus of presentation transcripts of ted talks. Procedia: Social and Behavioral Sciences, 24:174–182. find
  13. Jet Hoek, Jacqueline Evers-Vermeul, and Ted J.M. Sanders. 2019. Using the cognitive approach to coherence relations for discourse annotation. Dialogue & Discourse, 10(2):1–33. find
  14. Jet Hoek and Merel C.J. Scholman. 2017. Evaluating discourse annotation: Some recent insights and new approaches. In Proceedings of the 13th Joint ISO-ACL Workshop on Interoperable Semantic Annotation (ISA-13), pages 1–13, Toulouse, France. find
  15. Jet Hoek, Sandrine Zufferey, Jacqueline Evers-Vermeul, and Ted J M Sanders. 2017. Cognitive complexity and the linguistic marking of coherence relations: A parallel corpus study. Journal of Pragmatics, 121:113–131. find
  16. Eleni Miltsakaki, Aravind Joshi, Rashmi Prasad, and Bonnie Webber. 2004. Annotating discourse connectives and their arguments. In Proceedings of the Workshop Frontiers in Corpus Annotation at HLT-NAACL 2004, pages 9–16, Boston, MA, USA. find our card
  17. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC), pages 2961–2968, Marrakech, Marocco. find our card
  18. Ines Rehbein, Merel C. J. Scholman, and Vera Demberg. 2016. Annotating discourse relations in spoken language: A comparison of the PDTB and CCR frameworks. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC), pages 23–28, Portoroz, Slovenia. find our card
  19. Livio Robaldo and Eleni Miltsakaki. 2014. Corpus-driven semantics of concession: Where do expectations come from? Dialogue & Discourse, 5(1):1– 36. find
  20. Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie Webber. 2018. Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257–2267. find our card
  21. Ted J. M. Sanders, Vera Demberg, Jet Hoek, Merel C. J. Scholman, Fatemeh Torabi Asr, Sandrine Zufferey, and Jacqueline Evers-Vermeul. 2018. Unifying dimensions in discourse relations: How various annotation frameworks are related. Corpus Linguistics and Linguistic Theory, ahead of print:1–71. find
  22. Ted J. M. Sanders and Leo G. M. Noordman. 2000. The role of coherence relations and their linguistic markers in text processing. Discourse Processes, 29(1):37–60. find
  23. Ted J.M. Sanders, Wilbert P.M.S. Spooren, and Leo G.M. Noordman. 1992. Toward a taxonomy of coherence relations. Discourse Processes, 15(1):1– 35. find
  24. Merel C. J. Scholman and Vera Demberg. 2017. Examples and specifications that prove a point: Identifying elaborative and argumentative discourse relations. Dialogue & Discourse, 8(2):56–83. find our card
  25. Wilbert Spooren. 1997. The processing of underspecified coherence relations. Discourse Processes, 24(1):149–168. find
  26. Wilbert P.M.S. Spooren and Liesbeth Degand. 2010. Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory, 6(2):241– 266. find
  27. Deniz Zeyrek, Amália Mendes, Yulia Grishina, Murathan Kurfalı, Samuel Gibbon, and Maciej Ogrodniczuk. 2019. Ted multilingual discourse bank (TED-MDB): a parallel corpus annotated in the PDTB style. Language Resources and Evaluation, pages 1–27. find

↑ back to top