Elamur Research
☰
home / papers / 2014 / Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

Summary

Coreference metrics such as MUC, B-cubed and CEAF were originally defined for gold mentions, and different research groups adapted them to predicted mentions in incompatible ways, often by adding or removing mentions artificially so that the key and response partitions line up. This made published scores hard to compare. The paper states how each metric should be computed directly on partitions with predicted mentions, without such manipulations: a mention present in only one partition simply contributes nothing where it has no counterpart. It walks through worked examples and describes a reference implementation of the scorer that became the standard tool for CoNLL-style evaluation. The aim is a single, transparent definition so that results across systems and papers are comparable.

How Mova uses it

Effectiveness in Mova

Not an idea that changes scores, so not measured as a gain. Its role is correctness of the measuring instrument: the scorer reproduces the paper's worked example, and every reported coreference result (for example CoNLL F1 73.04 with singletons and 60.31 without on GUM test, gold trees) depends on it.

👨‍🔬💥

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P14-2006.pdf

Summary

This short paper addresses an inconsistency in coreference evaluation. When a system predicts its own mentions, those mentions rarely match the gold (key) mentions exactly. The original definitions of B3 and CEAF don't say how to handle that mismatch, and BLANC was built only for key mentions. Several variants tried to fix this by adding or removing "twinless" mentions on the key side or the response side, including Stoyanov et al. (2009), Rahman and Ng (2009) and Cai and Strube (2010). The CoNLL-2011/2012 scorer used the Cai and Strube approach, and its implementation of it was buggy. The authors argue that this mention manipulation is unnecessary and can reward bad output, because the original metrics already work on predicted mentions. They release an open-source, thoroughly tested scorer that follows the original metric definitions and includes the extension of BLANC to predicted mentions. They walk through every metric on one example and rescore all CoNLL-2011/2012 closed-track systems for English, Chinese and Arabic. This gives later end-to-end coreference work a trustworthy common baseline to compare against.

Key points

  • The scorer covers MUC, B3, CEAF_m, CEAF_e and BLANC. The CoNLL score stays the arithmetic mean of MUC, B3 and CEAF_e. The release fixes the CoNLL scorer v4 bugs reported by Recasens et al. (2013) and ships all of its test cases.
  • In a case adapted from Cai and Strube (2010), the key entity is {a,b,c} and the response is {a,b,d}:
    • B3_0 gives a perfect precision of 1.00, and B3_all gives 0.556.
    • If c is added as a spurious singleton, B3_all precision rises to 0.667.
    • The unmanipulated B3 sensibly falls from 0.444 to 0.333.
  • On a worked example with a missing mention and spurious mentions, the F1 scores are MUC 0.40, B3 0.46, CEAF_m 0.53 and CEAF_e 0.52. BLANC is 0.36.
  • After rescoring, CoNLL system rankings stayed mostly the same, with a few lower-ranked systems moving by one or two places. B3 scores dropped noticeably, which lowered the CoNLL averages. The drop comes from two things together: the old manipulation added singletons for twinless mentions, and B3 lets one entity be credited more than once.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2006

@inproceedings{pradhan2014scoring,
  title = {{Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation}},
  author = {Pradhan, Sameer and Luo, Xiaoqiang and Recasens, Marta and Hovy, Eduard and Ng, Vincent and Strube, Michael},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2006},
  url = {https://doi.org/10.3115/v1/p14-2006}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of LREC, pages 563–566. find our card
  2. Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of SIGDIAL, pages 28–36. find
  3. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the Sixth IJCNLP, pages 1366–1374, Nagoya, Japan, October. find
  4. Nancy Chinchor and Beth Sundheim. 2003. Message understanding conference (MUC) 6. In LDC2003T13. find
  5. Nancy Chinchor. 2001. Message understanding conference (MUC) 7. In LDC2001T02. find
  6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program-tasks, data, and evaluation. In Proceedings of LREC. find
  7. Lynette Hirschman and Nancy Chinchor. 1997. Coreference task definition (v3.0, 13 jul 97). In Proceedings of the 7th Message Understanding Conference. find
  8. Gordana Ilic Holen. 2013. Critical reflections on evaluation practices in coreference resolution. In Proceedings of the NAACL-HLT Student Research Workshop, pages 1–7, Atlanta, Georgia, June. find
  9. Daniel Jurafsky and James H. Martin. 2008. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall. Second Edition. find
  10. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An extension of BLANC to system mentions. In Proceedings of ACL, Baltimore, Maryland, June. find
  11. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP, pages 25–32. find our card
  12. Sameer Pradhan, Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. OntoNotes: A Unified Relational Semantic Representation. International Journal of Semantic Computing, 1(4):405–419. find
  13. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–27. find our card
  14. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–40. find our card
  15. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of EMNLP, pages 968–977. find
  16. Altaf Rahman and Vincent Ng. 2011. Coreference resolution with world knowledge. In Proceedings of ACL, pages 814–824. find
  17. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. find
  18. Marta Recasens, Lluı́s Màrquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, pages 1–8. find
  19. Marta Recasens, Marie-Catherine de Marneffe, and Chris Potts. 2013. The life and death of discourse entities: Identifying singleton mentions. In Proceedings of NAACL-HLT, pages 627–633. find
  20. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP, pages 656–664. find our card
  21. William F. Styler, Steven Bethard an Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014. Temporal annotation in the clinical domain. Transactions of Computational Linguistics, 2(April):143–154. find
  22. Ozlem Uzuner, Andreea Bodnari, Shuying Shen, Tyler Forbush, John Pestian, and Brett R South. 2012. Evaluating the state of the art in coreference resolution for electronic medical records. Journal of American Medical Informatics Association, 19(5), September. find
  23. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference, pages 45–52. find our card
  24. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer. find