Elamur Research
☰
home / papers / 2010 / Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information

Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information

What it is about

The paper shows that a coreference score says as much about the corpus as about the system. The same entity-based system, CISTELL, is trained and tested under three corpus choices: language (English against Spanish, using OntoNotes and AnCora, both news), coding scheme (ACE against OntoNotes on the same 100,000-word TDT portion), and preprocessing (gold against automatic parses). The system uses four basket-growing strategies (head match, strong match, best match, weak match) and a memory-based classifier. The scores are MUC, B3 and CEAF. The rankings of the strategies disagree across the metrics: MUC rewards weak match most, B3 rewards head match most, and CEAF is split between super-strong and best match. In both datasets about 85% of entities are singletons and 15% have two or more mentions, with 3.9 mentions per multi-mention entity in AnCora and 3.5 in OntoNotes. The authors conclude that system comparison is only meaningful when corpus parameters match exactly.

Ideas

For Mova

Code: coref/src/eval.rs (three metrics side by side), coref/src/sieve.rs (variants as separate configurations), coref/src/corefud.rs (scheme conversion). Chapter: languages/en/35-reference-and-coreference.md. Measure: the sieve under two scheme settings and three metrics, with the rankings printed; run twice and compare the output files byte for byte.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P10-1144.pdf

Summary

The paper asks how much a coreference resolver's reported score depends on the corpus rather than on the system. Recasens and Hovy keep a single system fixed: CISTELL, an entity-based resolver that grows "baskets" of attributes for discourse entities and uses a TiMBL memory-based pairwise classifier. They then vary three corpus parameters one at a time. Language is varied by comparing English with Spanish (AnCora). Annotation scheme is varied by comparing ACE and OntoNotes on a shared 100k-word TDT portion. Preprocessing is varied by comparing gold-standard and automatic linguistic information. Every configuration includes three baselines and four clustering models, all run on true mentions and scored with MUC, B3 and CEAF. Scores shift substantially with each parameter, and the three metrics show systematic biases that often rank the models in contradictory orders. The authors conclude that two systems can be compared fairly only when they use the same corpus, the same preprocessing and the same measures, and all measures agree. This matters for anyone who benchmarks coreference systems or builds annotated corpora.

Key points

  • Setup: There are three baselines (ALL SINGLETONS, HEAD MATCH, HEAD MATCH + PRON) and four CISTELL models (STRONG, SUPER STRONG, BEST and WEAK MATCH). SUPER STRONG MATCH accepts a link only when the TiMBL distance is below 0.09. English uses 29 pairwise features; Spanish uses 34, partly to handle zero subjects.
  • Aligned data: The ACE/OntoNotes aligned portion has 297 training and 33 test documents, about 80.8k and 9.1k words respectively.
  • Preprocessing effect: On ACE, automatic preprocessing lowered scores by roughly 3% compared with gold OntoNotes information. Dropping the noisy grammatical-function feature then raised results by about 2–3%.
  • Metric biases:
    • MUC is too forgiving of spurious links.
    • B3 is inflated when a corpus contains many singletons.
    • CEAF disagrees with both.
    • In one table, the model B3 ranks highest is the one MUC ranks lowest.
  • External support: In SemEval-2010 Task 1, even the strongest systems struggled to beat the all-singletons baseline.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{recasens2010coreference,
  title = {{Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information}},
  author = {Recasens, Marta and Hovy, Eduard},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1144/}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC 1998 Workshop on Linguistic Coreference, pages 563–566, Granada, Spain. find our card
  2. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of EMNLP 2008, pages 294–303, Honolulu, Hawaii. find our card
  3. Thorsten Brants. 2000. TnT – A statistical part-of-speech tagger. In Proceedings of ANLP 2000, Seattle, WA. find our card
  4. Aron Culotta, Michael Wick, Robert Hall, and Andrew McCallum. 2007. First-order probabilistic models for coreference resolution. In Proceedings of HLT-NAACL 2007, pages 81–88, Rochester, New York. find
  5. Walter Daelemans and Antal Van den Bosch. 2005. Memory-Based Language Processing. Cambridge University Press. find
  6. Pascal Denis and Jason Baldridge. 2009. Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, 42:87–96. find
  7. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) Program - Tasks, Data, and Evaluation. In Proceedings of LREC 2004, pages 837–840. find
  8. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. The MIT Press. find
  9. Jenny Rose Finkel and Christopher D. Manning. 2008. Enforcing transitivity in coreference resolution. In Proceedings of ACL-HLT 2008, pages 45– 48, Columbus, Ohio. find
  10. Aria Haghighi and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of EMNLP 2009, pages 1152–1161, Singapore. Association for Computational Linguistics. find our card
  11. Johan Hall, Jens Nilsson, Joakim Nivre, Gülsen Eryigit, Beáta Megyesi, Mattias Nilsson, and Markus Saers. 2007. Single malt or blended? A study in multilingual parser optimization. In Proceedings of the CoNLL shared task session of EMNLP-CoNLL 2007, pages 933–939. find
  12. Lynette Hirschman and Nancy Chinchor. 1997. MUC-7 Coreference Task Definition – Version 3.0. In Proceedings of MUC-7. find
  13. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of HLT-NAACL 2006, pages 57–60. find
  14. Taku Kudoh and Yuji Matsumoto. 2000. Use of support vector learning for chunk identification. In Proceedings of CoNLL 2000 and LLL 2000, pages 142– 144, Lisbon, Portugal. find
  15. Xiaoqiang Luo and Imed Zitouni. 2005. Multi-lingual coreference resolution with syntactic features. In Proceedings of HLT-EMNLP 2005, pages 660–667, Vancouver. find
  16. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In Proceedings of ACL 2004, pages 21–26, Barcelona. find
  17. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP 2005, pages 25–32, Vancouver. find our card
  18. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of ACL 2002, pages 104–111, Philadelphia. find our card
  19. Vincent Ng. 2009. Graph-cut-based anaphoricity determination for coreference resolution. In Proceedings of NAACL-HLT 2009, pages 575–583, Boulder, Colorado. find
  20. Hoifung Poon and Pedro Domingos. 2008. Joint unsupervised coreference resolution with Markov logic. In Proceedings of EMNLP 2008, pages 650–659, Honolulu, Hawaii. find
  21. Sameer S. Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. Ontonotes: A unified relational semantic representation. In Proceedings of ICSC 2007, pages 517–526, Washington, DC. find
  22. Marta Recasens and Eduard Hovy. 2009. A Deeper Look into Features for Coreference Resolution. In S. Lalitha Devi, A. Branco, and R. Mitkov, editors, Anaphora Processing and Applications (DAARC 2009), volume 5847 of LNAI, pages 29–42. Springer-Verlag. find
  23. Marta Recasens and M. Antònia Martı́. 2009. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, DOI 10.1007/s10579-009-9108-x. ↗
  24. Marta Recasens, Lluı́s Màrquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 Task 1: Coreference resolution in multiple languages. In Proceedings of the Fifth International Workshop on Semantic Evaluations (SemEval 2010), Uppsala, Sweden. find
  25. Wee M. Soon, Hwee T. Ng, and Daniel C. Y. Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. find our card
  26. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP 2009, pages 656–664, Singapore. find our card
  27. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-independent Named Entity Recognition. In Walter Daelemans and Miles Osborne, editors, Proceedings of CoNLL 2003, pages 142–147. Edmonton, Canada. find
  28. Olga Uryupina. 2006. Coreference resolution with and without linguistic knowledge. In Proceedings of LREC 2006. find
  29. Kees van Deemter and Rodger Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4):629– 637. find our card
  30. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, pages 45–52, San Francisco. find our card