Elamur Research
☰
home / papers / 2000 / On Coreferring: Coreference in MUC and Related Annotation Schemes

On Coreferring: Coreference in MUC and Related Annotation Schemes

What it is about

The paper argues that MUC-style "coreference" annotation covers more than the coreference relation (two noun phrases with the same referent). It separates coreference from anaphora (anaphora is irreflexive, non-symmetric and non-transitive, and depends on context; coreference is an equivalence relation), and it shows where the MUC guidelines ask annotators to link things that do not corefer: non-referring NPs in "No solution emerged ... or ... whenever a solution emerged ... we embraced it", bound anaphora such as "Every TV network reported its projects", and predicative NPs such as "a president of the company" linked to the subject. Their proposal is a division of labour: annotate the coreference relation proper in one layer, and bound anaphora and subject-predicate relations in separate layers, and rethink the task before it is extended to speech or to whole/part relations. The paper has no experiment; its argument rests on constructed examples and on the MUC task definition, which lists four priorities (support for the extraction tasks, about 95% inter-annotator agreement, cheap annotation, a corpus usable by outside linguists).

Ideas

For Mova

Code: coref/src/sieve.rs (link types), coref/src/corefud.rs (annotation layers), prag (bound and predicative readings for prag/src/frames.rs). Chapter: languages/en/35-reference-and-coreference.md, with languages/en/23-complementation.md for the predicative case. Measure: a test set of 40 constructed sentences (bound, predicative, identity), counted by type, with the sieve output checked by hand; run twice.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J00-4005.pdf

Summary

This squib examines what corpora labelled for "coreference", above all those built for MUC-6 and MUC-7, actually encode. The authors define coreference strictly: two NPs corefer only when both have a referent and the referents are identical. They contrast this with anaphora, where one expression depends on another for its interpretation. Measured against this definition, the MUC Task Definition (TD) asks annotators to link many items that are not coreferent: non-referring and quantified NPs, bound anaphors, predicative NPs, and descriptions that hold only at different times. Because coreference is treated as an equivalence relation, these links yield wrong conclusions, such as a former sales director and a current president being the same person. The relation MUC really annotates, called IDENT, therefore has no coherent motivation. The paper offers no new data or system. It argues that the TD meets its own goals only partly and calls for rethinking the task: annotate genuine coreference from a clear definition, and handle bound anaphora, predication and time-dependent properties in separate annotation tasks.

Key points

  • The MUC TD (Hirschman and Chinchor 1997) sets four ranked criteria:
    • support for information extraction,
    • about 95% interannotator agreement,
    • fast and cheap annotation,
    • a corpus useful to linguists outside MUC.
  • Reported interannotator agreement was in the low eighties, well short of 95%, and reportedly had not improved much since. MUC information-extraction recall was also weak (Appelt 1998).
  • The TD's instruction to mark NPs as coreferential if linked "at any time" makes chains like Henry Higgins / sales director of Sudsy Soaps / president of Dreamy Detergents incoherent. Epistemic modality (might be the man…) raises analogous problems.
  • Linking Every TV network with its under the coreference definition gives the wrong reading of the sentence. Predicative NPs such as the president of DD also fail a substitution test with the name they are linked to.
  • The authors propose two options for deciding which NPs count as referring:
    • a strict one based on semantically definite NPs (Barwise and Cooper 1981), which is easier and more reliable to annotate;
    • a liberal one following Kamp and Reyle (1993), which feeds more information to information extraction.
  • The authors note that the trade-off between these options must be settled by experiment.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/089120100750105966

@article{deemter2000coreferring,
  title = {{On Coreferring: Coreference in MUC and Related Annotation Schemes}},
  author = {Deemter, Kees van and Kibble, Rodger},
  journal = {Computational Linguistics},
  year = {2000},
  doi = {10.1162/089120100750105966},
  url = {https://doi.org/10.1162/089120100750105966}
}

References

27 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ABPI. 1997. 1996-1997 ABPI Compendium of Patient Information Leaflets. Association of the British Pharmaceutical Industry. find
  2. Appelt, Douglas. 1998. An overview of information extraction technology and its application to information retrieval. In Proceedings of TWLT14, Language Technology in Multimedia Information Retrieval, pages 49-58, Twente. find
  3. Bach, Kent. 1987. Thought and Reference. Clarendon Press, Oxford. find
  4. Bagga, Amit, Breck Baldwin, and Sara Shelton. 1998. Coreference and its applications. Call for papers for workshop associated with the 37th Annual Meeting of the Association for Computational Linguistics, University of Maryland, 1999. See www.cs.duke.edu/,-~amit / acc99-wkshp.html. find
  5. Barwise, Jon and Robin Cooper. 1981. Generalized quantifiers and natural language. Linguistics and Philosophy, 4:159-219. find
  6. Cristea, Dan, Nancy Ide, Daniel Marcu, and Valentin Tablan. 1999. Discourse structure and coreference: An empirical study. In Dan Cristea, Nancy Ide, and Daniel Marcu, editors, Proceedings of ACL'99 Ws: The Relation of Discourse~Dialogue Structure and Reference, pages 46-53. find
  7. Davies, Sarah, Massimo Poesio, Florence Bruneseaux, and Laurent Romary. 1998. Annotating coreference in dialogues: Proposal for a scheme for MATE. See www.cogsci.ed.ac.uk/~poesio/MATE/anno-manual.html. find
  8. Donnellan, Keith. 1966. Reference and definite descriptions. Philosophical Review, 75:281-304. find
  9. Dowty, David, Robert Wall, and Stanley Peters. 1981. Introduction to Montague Semantics. Kluwer, Dordrecht. find
  10. Gamut, L. T. F. 1991. Logic, Language and Meaning, Volume 2. University of Chicago Press, Chicago. find
  11. Garside, Roger, Geoffrey Leech, and Tony McEnery. 1997. Corpus Annotation. Longman, London. find
  12. Groenendijk, Jeroen, Martin Stokhof, and Frank Veltman. 1996. Coreference and modality. In Shalom Lappin, editor, The Handbook of Contemporary Semantic Theory. Blackwell, Cambridge, MA, pages 179-214. find
  13. Hirschman, Lynette and Nancy Chinchor. 1997. MUC-7 coreference task definition. In MUC-7 Proceedings. Science Applications International Corporation. See www.muc.saic.com. find
  14. Hirschman, Lynette, Patricia Robinson, John Burger, and Marc Vilain. 1997. Automating coreference: The role of annotated training data. In Proceedings of AAAI Spring Symposium on Applying Machine Learning to Discourse Processing. find
  15. Kamp, Hans and Uwe Reyle. 1993. From Discourse to Logic. Kluwer, Dordrecht. find
  16. Kibble, Rodger and Kees van Deemter. 2000. Coreference annotation: Whither? In Maria Gavrilidou et al., editors, Proceedings of the 2nd International Conference on Language Resources and Evaluation, pages 1,281-1,286, Athens. find
  17. Kronfeld, Ami and Lawrence Roberts. 1998. Special Issue on Reference, Pragmatics and Cognition 6. John Benjamins, Amsterdam and Philadelphia. find
  18. Partee, Barbara. 1989. Binding implicit variables in quantified contexts. Proceedings of the Chicago Linguistic Society, 25:342-365. find
  19. Partee, Barbara, Alice ter Meulen, and Robert Wall. 1990. Mathematical Methods in Linguistics. Kluwer, Dordrecht. find
  20. Passoneau, Rebecca. 1997. Instructions for applying discourse reference annotation for multiple applications (DRAMA). Unpublished manuscript. find
  21. Poesio, Massimo. 2000. Annotating a corpus to develop and evaluate discourse entity realization algorithms: Issues and preliminary results. In Maria Gavrilidou et al., editors, Proceedings of the 2nd International Conference on Language Resources and Evaluations, pages 211-218, Athens. find
  22. Popescu-Belis, Andrei. 1998. How corpora with annotated coreference links improve reference resolution. In Antonio Rubio et al., editors, First International Conference on Language Resources and Evaluation, pages 567-572, Granada. European Language Resources Association. find
  23. Sidner, Candace. 1979. Towards a Computational Theory ofDeJinite Anaphora Comprehension in English Discourse. Ph.D. dissertation, AI Lab, MIT, Cambridge, MA. find
  24. Sidner, Candace. 1983. Focusing in the comprehension of definite anaphora. In Michael Brady and Robert Berwick, editors, Computational Models of Discourse. MIT Press, Cambridge, MA, pages 267-330. find
  25. van Deemter, Kees. 1992. Towards a generalization of anaphora. Journal of Semantics, 9:27-51. find
  26. van Deemter, Kees and Rodger Kibble. 1999. What is coreference and what should coreference annotation be? In Amit Bagga, Breck Baldwin, and Sara Shelton, editors, Proceedings of ACL workshop on Coreference and Its Applications, pages 90-96, Maryland. find
  27. van der Sandt, Rob. 1992. Presupposition projection as anaphora resolution. Journal of Semantics, 9:333-37. find