Elamur Research⌕
☰
home / papers / 2005 / The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account

The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account

Summary

Poesio and Artstein revisit how reliable anaphoric annotation is. They argue that earlier agreement studies used too few coders or lacked a suitable chance-corrected statistic. They also argue that disagreement often reflects real ambiguity in the text rather than a flawed scheme. To test this, eighteen naive, paid students annotated a TRAINS 91 dialogue in MMAX 2. The coding manual was loosely derived from MATE and GNOME. It let coders flag an expression as ambiguous and mark simple discourse deixis, and it left out bridging references.

Agreement was measured with Krippendorff's α rather than K, following Passonneau, because α gives credit for partial overlap between anaphoric chains. The authors compared three chain distance metrics (Passonneau's, Jaccard, Dice) and several ways of building chains from the raw annotations. The chain-construction method affected α more than the choice of metric. Qualitative analysis showed that almost all disagreements, apart from occasional careless errors, came from genuine referential ambiguity. This matters for designing corpora and for future methods that detect ambiguous expressions automatically.

Key points

  • Setup: 18 University of Essex students, mostly undergraduates in Psychology and Language and Linguistics, annotated dialogue 3.2 of TRAINS 91 in MMAX 2. They trained on dialogue 3.1, and the work was split over two sessions five days apart.
  • Conditions compared: six chain conditions (no chain, partial chain, and full chains in inclusive/exclusive and [+top]/[−top] variants), each crossed with three distance metrics.
  • Best result: the Inclusive [−top] chain with the Dice distance gave the highest α. For the first half of the dialogue, α was about 0.69 with place markables and about 0.58 without them.
  • Careless errors: obvious mistakes accounted for only 35 of 1296 judgments (about 2.7%).
  • Ambiguity example: for the pronoun "it" in utterance 3.1, 9 coders flagged it as ambiguous, 6 chose engine E2 and 3 chose the boxcar. This shows that implicit ambiguity, revealed by disagreement between coders, matters as much as ambiguity coders mark explicitly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 46 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{poesio2005reliability,
  title = {{The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account}},
  author = {Poesio, Massimo and Artstein, Ron},
  booktitle = {Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky},
  year = {2005},
  url = {https://aclanthology.org/W05-0311/}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Buitelaar. 1998. CoreLex : Systematic Polysemy and Underspecification. Ph.D. thesis, Brandeis University. find
  2. D. Byron. 2003. Annotation of pronouns and their antecedents: A comparison of two domains. Technical Report 703, University of Rochester. find
  3. M. Eckert and M. Strube. 2001. Dialogue acts, synchronising units and anaphora resolution. Journal of Semantics. find
  4. D. Gross, J. Allen, and D. Traum. 1993. The TRAINS 91 dialogues. TRAINS Technical Note 92-1, Computer Science Dept. University of Rochester, June. find
  5. L. Hirschman. 1998. MUC-7 coreference task definition, version 3.0. In N. Chinchor, editor, In Proc. of the 7th Message Understanding Conference. find our card
  6. H. Kamp and U. Reyle. 1993. From Discourse to Logic. D. Reidel, Dordrecht. find
  7. K. Krippendorff. 1980. Content Analysis: An introduction to its Methodology. Sage Publications. find
  8. C. D. Manning and H. Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press. find
  9. C. Müller and M. Strube. 2003. Multi-level annotation in MMAX. In Proc. of the 4th SIGDIAL. find
  10. M. Palmer, H. Dang, and C. Fellbaum. 2005. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Journal of Natural Language Engineering. To appear. find
  11. R. J. Passonneau. 1997. Instructions for applying discourse reference annotation for multiple applications (DRAMA). Unpublished manuscript., December. find
  12. R. J. Passonneau. 2004. Computing reliability for coreference annotation. In Proc. of LREC, Lisbon. find
  13. M. Poesio and R. Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183–216, June. find
  14. M. Poesio, F. Bruneseaux, and L. Romary. 1999. The MATE meta-scheme for coreference in dialogues in multiple languages. In M. Walker, editor, Proc. of the ACL Workshop on Standards and Tools for Discourse Tagging, pages 65–74. find our card
  15. M. Poesio. 1996. Semantic ambiguity and perceived ambiguity. In K. van Deemter and S. Peters, editors, Semantic Ambiguity and Underspecification, chapter 8, pages 159–201. CSLI, Stanford, CA. find
  16. M. Poesio. 2004. The MATE/GNOME scheme for anaphoric annotation, revisited. In Proc. of SIGDIAL, Boston, May. find our card
  17. E. F. Prince. 1992. The ZPG letter: subjects, definiteness, and information status. In S. Thompson and W. Mann, editors, Discourse description: diverse analyses of a fund-raising text, pages 295–325. John Benjamins. find
  18. A. Rosenberg and E. Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proc. of NAACL. find
  19. C. L. Sidner. 1979. Towards a computational theory of definite anaphora comprehension in English discourse. Ph.D. thesis, MIT. find
  20. S. Siegel and N. J. Castellan. 1988. Nonparametric statistics for the Behavioral Sciences. McGraw-Hill. find
  21. K. van Deemter and R. Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4):629–637. Squib. find our card
  22. B. L. Webber. 1979. A Formal Approach to Discourse Anaphora. Garland, New York. find
  23. B. L. Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107–135. find

↑ back to top