Elamur Research⌕
☰
home / papers / 2013 / Beyond semantics: the challenges of annotating pragmatic and discourse phenomena

Beyond semantics: the challenges of annotating pragmatic and discourse phenomena

Paper · 2013 · Stefanie Dipper, Heike Zinsmeister and Bonnie Webber

Summary

This piece is the editors' introduction to a Dialogue & Discourse special issue on annotating meaning that lies beyond sentence semantics: discourse relations, coherence, dialogue acts, specificity, inference-triggering verb relations and information structure. The editors argue that such phenomena are rarely marked explicitly in text. Annotators must reconstruct them from context, so agreement is often weak, and theories built on invented examples transfer poorly to real corpora. They survey a common remedy found across the collected papers: replacing an abstract judgement with a more tangible proxy. Examples include inserting a connective, rephrasing with a test frame, using translation equivalents, following morphological cues, or answering a question-based decision tree. The issue gathers twelve papers on corpora for German, English, Turkish, Danish and other languages, together with classifiers trained on them. The editors' broader aim is to bring theoretical linguists and computational annotators closer together. Operationalized concepts should yield more reliable annotation, corpus-based testing of hypotheses, and better use of pragmatic knowledge in NLP.

Key points

  • Earlier work (Ritz et al., 2008) found that annotating information structure in German often produces Cohen's κ well below 0.6. Artstein and Poesio (2008) report similarly low κ for dialogue acts, discourse segmentation and word sense tagging.
  • Penn Discourse Treebank practice serves as the model proxy: for implicit relations, annotators supply an overt connective. Versley & Gastel extend this in TüBa-D/Z with substitution and insertion tests, paraphrases, nominalizations and explicit questions under discussion. Their scheme organizes relations into five classes: temporal, comparison, contingency, expansion and reporting.
  • Cartoni, Zufferey & Meyer approach connective senses through "translation spotting" in a parallel corpus. They train a Maximum Entropy classifier that separates six senses of English while.
  • Several resources adapt or build on PDTB-style or existing annotation:
    • The Turkish Discourse Bank adapts PDTB-style annotation and treats nominalizations as discourse arguments.
    • The Copenhagen Dependency Treebanks place discourse annotation on top of dependency syntax instead of raw strings.
    • Taboada & Das add signalling cues to the existing RST Discourse Treebank.
  • Louis & Nenkova found that 64% of sentences received the same general/specific label from at least four judges. Their classifier's confidence tracked human agreement. In total, the call drew 32 abstracts and 16 submissions, of which 12 papers were accepted (10 regular papers and 2 short notes).

Interesting ideas

  • I1. Swap a vague label for an operational test that annotators can actually perform, such as inserting a connective or rewriting a sentence as "Concerning X, …" to check whether X is the topic; this makes hidden judgements checkable and repeatable.
  • I2. Use a cross-lingual translation of an ambiguous word as evidence of its sense, since another language often splits senses that English merges (concessive versus contrastive versus temporal while).
  • I3. Separate lexical givenness (a word repeated) from referential givenness (the same entity mentioned again), as RefLex does; the two layers do not always coincide and both bear on focus.
  • I4. Anchor discourse relations to dependency structure rather than to character spans, as the Copenhagen Dependency Treebanks do; argument boundaries then follow syntactic heads.
  • I5. Treat classifier confidence as a signal of how prototypical an item is, since items on which humans agree tend to be the ones a model is sure about.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could decide implicit discourse relations and aboutness topics by trying candidate connectives or topic frames on the parsed clause pair, and record which test succeeded as the explanation.
  • I2 (learn): translation equivalents from parallel text could act as an additional witness when building silver sense labels for connectives, with disagreements flagged for the LLM teacher.
  • I3 (world): entity tracking could keep the two layers apart, so that "a man… the man" marks coreference while a mere repeated noun does not wrongly merge two entities.
  • I4 (tree): discourse links could be attached to clause heads in the UD tree, letting relation arguments be read straight off subtrees and keeping the tree-to-frames handoff explainable.
  • I5 (learn): when evaluating, low-confidence or witness-disagreement items can be split out as a likely non-prototypical set, rather than being counted as plain errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓Universität HamburgHamburg · Germany39 works · 4,669 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Discourse analysis · 394 works

Cite

Cite as Stefanie Dipper, Heike Zinsmeister and Bonnie Webber. 2013. Beyond semantics: the challenges of annotating pragmatic and discourse phenomena. Dialogue & Discourse Volume 4.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5210/dad.2013.200

@inproceedings{dipper2013beyond,
  title = {{Beyond semantics: the challenges of annotating pragmatic and discourse phenomena}},
  author = {Stefanie Dipper, Heike Zinsmeister and Bonnie Webber},
  booktitle = {Dialogue \& Discourse Volume 4},
  year = {2013},
  doi = {10.5210/dad.2013.200},
  url = {https://doi.org/10.5210/dad.2013.200}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ACE-EDT. Annotation guidelines for entity detection and tracking (EDT). Linguistic Data Consortium, 2004. URL http://catalog.ldc.upenn.edu/docs/LDC2005T09/guidelines/EnglishEDTV4-2-6.PDF. Version 4.2.6 200400401. ↗
  2. Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Michael Kipp, Stephan Koch, Elisabeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. Dialogue acts in VERBMOBIL-2 (Second edition). Verbmobil Report 226, Saarland University, Saarbrücken, Germany, 1998. URL http://www.coli.uni-sb.de/publikationen/softcopies/Alexandersson:1998:DAV.pdf. ↗
  3. James Allen and Mark Core. Draft of DAMSL: Dialogue act markup in several layers. Technical report, University of Rochester, 1997. URL http://www.cs.rochester.edu/research/speech/damsl/RevisedManual/. ↗
  4. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jaqueline C. Kowtko, Jan McAllister, Jim Miller, Cathy Sotillo, Henry Thompson, and Regina Weinert. The HCRC Map Task Corpus. Language and Speech, 34(4): 351–366, 1991. ↗
  5. Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics (survey article). Computational Linguistics, 34(4):555–596, 2008. ↗ our card
  6. Ada Brunstein. Annotation guidelines for answer types. Technical report, BBN Technologies, 2002. URL http://catalog.ldc.upenn.edu/docs/LDC2005T33/BBN-Types-Subtypes.html. ↗
  7. Jean Carletta, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1):13–31, 1997. ↗ our card
  8. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Jan van Kuppevelt and Ronnie Smith, editors, Current and New Directions in Discourse and Dialogue, pages 85–112. Kluwer Academic Publishers, 2003. ↗ our card
  9. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. ↗
  10. VIII Stefanie Dipper and Heike Zinsmeister, editors. Beyond Semantics: Corpus-based Investigations of Pragmatic and Discourse Phenomena. Proceedings of the DGfS Workshop, Göttingen, volume 3 of BLA (Bochumer Linguistische Arbeiten). Institute of Linguistics; Ruhr-University Bochum, 2011. URL http://www.linguistics.ruhr-uni-bochum.de/bla/. ↗
  11. Charles Fillmore, Josef Ruppenhofer, and Collin F. Baker. FrameNet and representing the link between semantic and syntactic relations. In Chu-Ren Huang and Winfried Lenders, editors, Computational Linguistics and Beyond, Language and Linguistics Monographs Series B. Frontiers in Linguistics I, pages 19–62. Institute of Linguistics, Academia Sinica, Taipei, 2004. find
  12. Michael Götze, Thomas Weskott, Cornelia Endriss, Ines Fiedler, Stefan Hinterwimmer, Svetlana Petrova, Anne Schwarz, Stavros Skopeteas, and Ruben Stoel. Information Structure. In Stefanie Dipper, Michael Götze, and Stavros Skopeteas, editors, Information Structure in Cross-Linguistic Corpora, number 7 in Interdisciplinary Studies on Information Structure (ISIS), pages 147–187. Universitätsverlag Potsdam, 2007. find
  13. Lynette Hirschman and Nancy Chinchor. MUC-7 coreference task definition—version 3.0. In Proceedings of the 7th Message Understanding Conference (MUC-7), Fairfax, Virginia, 1998. our card
  14. Dan Jurafsky, Liz Shriberg, and Debra Biasca. Switchboard SWBD-DAMSL shallow-discourse-function annotation. Coders manual, draft 13. Technical Report RT 97-02, University of Colorado at Boulder & SRI International, 1997. URL http://www.stanford.edu/˜jurafsky/ws97/manual.august1.html. ↗
  15. J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174, 1977. ↗
  16. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. The NomBank project: An interim report. In Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 24–31, Boston, Massachusetts, 2004. our card
  17. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. A semantic concordance. In Proceedings of the Workshop on Human Language Technology (HLT-93), pages 303–308, Stroudsburg, Pennsylvania, 1993. our card
  18. MUC-6. Named Entity task definition. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 317–322, Columbia, Maryland, 1995. URL http://aclweb.org/anthology/M/M95/M95-1024.pdf. Version 2.1. ↗
  19. Martha Palmer, Daniel Gildea, and Paul Kingsbury. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–105, 2005. ↗ our card
  20. Rebecca J. Passonneau. Instructions for Applying Discourse Reference Annotation for Multiple Applications (DRAMA). Unpublished. Department of Computer Science, Columbia University, 1996. find
  21. Massimo Poesio. MATE dialogue annotation guidelines: Coreference. MATE Deliverable D2.1, pages 134–187, 2000. URL http://www.andreasmengel.de/pubs/mdag.pdf. ↗
  22. IX Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC’08), pages 2961–2968, Marrakech, Morocco, 2008. our card
  23. James Pustejovsky, José Castaño, Robert Ingria, Roser Saurı́, Robert Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir Radev. TimeML: Robust specification of event and temporal expressions in text. In Mark T. Maybury, editor, New directions in Question Answering. Papers from the 2003 AAAI Spring Symposium, pages 28–34. find
  24. AAAI Press, Menlo Park, California, 2003a. find
  25. James Pustejovsky, Patrick Hanks, Roser Saurı́, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. The TIMEBANK Corpus. In Proceedings of Corpus Linguistics, pages 647–656, 2003b. find
  26. Julia Ritz, Stefanie Dipper, and Michael Götze. Annotation of information structure: An evaluation across different types of texts. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC’08), pages 2137–2142, Marrakech, Morocco, 2008. our card
  27. Yuping Zhou and Nianwen Xue. PDTB-style discourse annotation of Chinese text. In Proceedings the 50th Annual Meeting of the Association for Computational Linguistics (ACL’12), pages 69– 77, Jeju, South Korea, 2012. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dipper-2013-beyond-semantics-challenges-annotating-pragmatic/