Elamur Research⌕
☰
home / papers / 2012 / Annotating Near-Identity from Coreference Disagreements

Annotating Near-Identity from Coreference Disagreements

Summary

Standard coreference corpora treat coreference as a yes/no relation. Borderline cases, where two mentions refer to "almost" the same entity, therefore cause annotator disagreement and add noise for resolution systems. The authors extend two existing coreference resources, the English NP4E and the Catalan AnCora-CA, with near-identity labels on 50K-token subsets of each. In a first attempt, annotators marked an explicit identity degree (total, strong near-identity, weak near-identity) on existing links. Only 2–6% of relations received a near-identity label, which suggests people are not consciously aware of the phenomenon. The second attempt was indirect. Five annotators made only a binary coreferent/non-coreferent decision, and a merging algorithm turned their level of agreement into identity degrees. This approach classified 12–16% of relations as near-identical. The released corpora, NIdent-EN and NIdent-CA, are presented as the first resources annotated for near-identity. They can support coreference systems that separate clear identity from borderline cases, as well as linguistic research on reference.

Key points

  • Source data: NP4E contains Reuters newswire on terrorism and security across five topics. AnCora-CA is a 400K-word Catalan news corpus, of which a 50K-token subset was used. Both were converted to the AnCora-CA entity-ID format.
  • Merging algorithm: each link is scored by the share of the five annotators who marked it, and links with a score above 0.4 (at least 2 annotators) are kept. Mention scores above 0.5 count as clear coreference, scores of 0.25–0.5 as identity degree 2, and scores up to 0.25 as degree 1.
  • Implicit-method distribution:
    • NP4E: 88.19% total identity, 10.86% degree 2, 0.95% degree 1.
    • AnCora-CA: 83.75% total identity, 14.75% degree 2, 1.50% degree 1.
  • Near-identity rose from 6% to 12% in English and from 2% to 16% in Catalan. The authors relate the higher Catalan rate to Romance languages' preference for anaphors whose head differs from the previous mention.
  • NIdent-EN and NIdent-CA are freely available at clic.ub.edu/corpus/nident.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{recasens2012annotating,
  title = {{Annotating Near-Identity from Coreference Disagreements}},
  author = {Recasens, Marta and Martí, M. Antònia and Orasan, Constantin},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  url = {https://aclanthology.org/L12-1391/}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Brockett. 2007. Aligning the RTE 2006 Corpus. Technical Report MSR-TR-2007-77, Microsoft Research. find
  2. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) program – tasks, data, and evaluation. In Proceedings of LREC 2004, pages 837–840. find our card
  3. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Proceedings of HLT-NAACL 2010, pages 385–393. find our card
  4. Laura Hasler and Constantin Orasan. 2009. Do coreferential arguments make event mentions coreferential? In Proceedings of DAARC 2009, pages 151–163. find
  5. Laura Hasler, Constantin Orasan, and Karin Naumann. 2006. NPs for Events: Experiments in coreference annotation. In Proceedings of LREC 2006, pages 1167–1172. find our card
  6. Erhard Hinrichs, Sandra Kübler, and Karin Naumann. 2005. A unified representation for morphological, syntactic, semantic and referential annotations. In Proceedings of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 13–20. find
  7. Lynette Hirschman and Nancy Chinchor. 1997. MUC-7 Coreference Task Definition – Version 3.0. In Proceedings of MUC-7. find our card
  8. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 Shared Task. In Proceedings of CoNLL: Shared Task, pages 28–34. find our card
  9. Lita Lundquist. 2007. Lexical anaphors in Danish and French. In Monika Schwarz-Friesel, Manfred Consten, and Mareile Knees, editors, Anaphors in Text: Cognitive, formal and applied approaches to anaphoric reference, pages 37–48. John Benjamins, Amsterdam, Netherlands. find
  10. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP 2005, pages 25–32. find our card
  11. Rebecca Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of LREC 2004, pages 1503–1506. find
  12. Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proceedings of LREC 2008, pages 1170–1174. find our card
  13. Massimo Poesio. 2004. The MATE/GNOME proposals for anaphoric annotation, revisited. In Proceedings of SIG-DIAL, pages 154–162. find our card
  14. Sameer Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. OntoNotes: A Unified Relational Semantic Representation. In Proceedings of the International Conference on Semantic Computing (ICSC), pages 517–526. find
  15. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–27. find our card
  16. Marta Recasens and M. Antònia Martı́. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315– 345. find
  17. Marta Recasens, Eduard Hovy, and M. Antònia Martı́. 2010. A typology of near-identity relations for coreference (NIDENT). In Proceedings of LREC 2010, pages 149–156. find our card
  18. Marta Recasens, Eduard Hovy, and M. Antònia Martı́. 2011. Identity, Non-Identity, and Near-Identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. find
  19. Marta Recasens, Liliana Tolchinsky, and M. Antònia Martı́. in prep. Coreference is not always either/or: Psycholinguistic evidence of near-identity. find
  20. Tony Rose, Mark Stevenson, and Miles Whitehead. 2002. The Reuters Corpus Volume 1 - from Yesterday’s News to Tomorrow’s Language Resources. In Proceedings of LREC 2002, pages 827–833. find
  21. Pasi Tapanainen and Timo Järvinen. 1997. A non-projective dependency parser. In Proceedings of ANLP 1997, pages 64–71. find
  22. Mariona Taulé, M. Antònia Martı́, and Marta Recasens. 2008. AnCora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of LREC 2008, pages 96–101. find
  23. Kees van Deemter and Rodger Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4):629–637. find our card
  24. Kees van Deemter. 2010. Not Exactly: In Praise of Vagueness. Oxford University Press, USA. find
  25. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6:333–353. find
  26. Annie Zaenen. 2006. Mark-up barking up the wrong tree. Computational Linguistics, 32(4):577–580. find

↑ back to top