Elamur Research
☰
home / papers / 2005 / On Coreference Resolution Performance Metrics

On Coreference Resolution Performance Metrics

Summary

The paper criticises existing coreference metrics, notably MUC (which ignores singletons and rewards over-merging) and the ACE value, and proposes CEAF (Constrained Entity-Alignment F-measure). CEAF finds the best one-to-one alignment between key and response entities, computed with the Kuhn-Munkres (Hungarian) algorithm, under a chosen entity-similarity function. Recall and precision are the total similarity of the alignment normalised by the self-similarity of the key and of the response respectively. Two similarity functions give the mention-based variant (number of shared mentions) and the entity-based variant (a Dice-like overlap per entity pair). The paper shows that CEAF behaves more sensibly than MUC on degenerate outputs such as putting all mentions in one cluster. CEAF later became one of the three components of the CoNLL coreference score.

How Mova uses it

Effectiveness in Mova

It is an evaluation metric, not a modelling idea, so it has no "gain" of its own. It is part of every coref measurement, e.g. the default system's CEAF-e F1 on GUM test is 74.68 with singletons and 52.89 without on gold trees (the development notes of coref).

👨‍🔬💥

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/H05-1004.pdf

Summary

Scoring a coreference system means comparing its entities (clusters of mentions) against gold entities. The metrics in use in 2005 had known flaws. The link-based MUC F-measure ignores singleton entities and rewards systems that merge too much. B-cubed can credit the same entity more than once. The ACE-value is a weighted cost that is hard to read as a share of correct output. Luo proposes CEAF (Constrained Entity-Alignment F-Measure), which pairs reference and system entities one-to-one so that the total similarity between pairs is as high as possible. Recall and precision come from dividing that total by the reference and system self-similarities. Finding the best pairing is a maximum bipartite matching problem, which the Kuhn-Munkres algorithm solves in polynomial time. CEAF comes in two forms: one based on mentions and one based on entities. Tests on MUC6 and ACE 2004 data show that the MUC F-measure keeps rising as a system outputs fewer entities. In contrast, CEAF peaks near the true entity count. This makes CEAF a symmetric score that is easier to interpret.

Key points

  • Because of the one-to-one constraint, extra system entities lower precision and missing ones lower recall. A perfect system scores 1, and a system with no output or no shared mentions scores 0.
  • There are two similarity functions. Counting shared mentions gives mention-based CEAF, which is roughly the share of mentions placed in the right entity. A per-pair F-measure gives entity-based CEAF, which is roughly the share of entities recognised correctly.
  • MUC6 test set: as the system's penalty setting varies, the MUC F-measure reaches its maximum (.902) with only 113 output entities. Mention-based CEAF peaks at 0.768 with 483 entities, much closer to the gold count, while at 113 entities it falls to 0.445.
  • ACE 2004 data: both ACE-value and CEAF penalise producing too many or too few entities. However, they peak at different settings: CEAF peaks at 0.794 with 930 entities, while ACE-value peaks at 89.7% with 1015–1105 entities. The two metrics correlate only weakly.
  • CEAF is symmetric and ACE-value is not, so CEAF suits comparisons between annotators where the order of inputs should not matter.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1220575.1220579

@inproceedings{luo2005coreference,
  title = {{On Coreference Resolution Performance Metrics}},
  author = {Luo, Xiaoqiang},
  booktitle = {Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  doi = {10.3115/1220575.1220579},
  url = {https://doi.org/10.3115/1220575.1220579}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation (LREC’98), pages 563–566. find our card
  2. Egon Balas, Donald Miller, Joseph Pekny, and Paolo Toth. 1991. A parallel shortest augmenting path algorithm for the assignment problem. Journal of the ACM (JACM), 38(4). find
  3. Francois Bourgeois and Jean-Claude Lassalle. 1971. An extension of the munkres algorithm for the assignment problem to rectangular matrices. Communications of the ACM, 14(12). find
  4. R. Fletcher. 1987. Practical Methods of Optimization. John Wiley and Sons. find
  5. Anshul Gupta and Lexing Ying. 1999. Algorithms for finding maximum matchings in bipartite graphs. Technical Report RC 21576 (97320), IBM T.J. Watson Research Center, October. find
  6. H.W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(83). find
  7. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proc. of ACL. find
  8. MUC-6. 1995. Proceedings of the Sixth Message Understanding Conference(MUC-6), San Francisco, CA. Morgan Kaufmann. find
  9. MUC-7. 1998. Proceedings of the Seventh Message Understanding Conference(MUC-7). find
  10. J. Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of SIAM, 5:32–38. find
  11. NIST. 2003a. The ACE evaluation plan. www.nist.gov/speech/tests/ace/index.htm. find
  12. NIST. 2003b. Proceedings of ACE’03 workshop. Booklet, Alexandria, VA, September. find
  13. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, , and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In In Proc. of MUC6, pages 45–52. find our card