Elamur Research⌕
☰
home / papers / 2010 / Evaluation Metrics For End-to-End Coreference Resolution Systems

Evaluation Metrics For End-to-End Coreference Resolution Systems

Summary

End-to-end coreference systems must detect mentions automatically, so their output contains "twinless" mentions: system mentions absent from the gold key, and gold mentions the system missed. Standard metrics assume key and response share the same mention set. Researchers have patched them in inconsistent and loosely documented ways, so published scores cannot be compared. Cai and Strube analyse MUC, B3 and CEAF and the existing B3 variants (B3 0 and B3 all from Stoyanov et al. 2009, and B3 r&n from Rahman & Ng 2009). Using small constructed examples, they show these variants are either too lenient or shift with the mention tagger rather than with resolution quality. They propose B3 sys and CEAF sys, which handle twinless mentions differently for precision and for recall. They then test the metrics with the BART reimplementation of Soon et al. (2001), fed by two mention taggers, and compare against Bengtson & Roth (2008). The new variants rank systems consistently with MUC and intuition, giving the field a reproducible basis for comparing end-to-end systems.

Key points

  • How B3 sys treats twinless mentions:
    • Missed gold mentions are inserted into the response as singletons.
    • Spurious system mentions left unresolved are dropped.
    • Spurious system mentions that were wrongly linked are added to the key, so they lower precision.
    • Recall is computed only over the original key chains.
  • Worked example (key {a b c} vs. response {a b d}):
    • B3 0 scores P 1.0 / F 0.615, because it ignores the spurious mention d.
    • B3 sys scores P 0.667 / R 0.556 / F 0.606.
  • Weakness of B3 all: it awards higher precision when a mention tagger merely extracts additional mentions, such as extra singletons, even if the coreference links are identical.
  • Realistic setting (Soon, SM2 tagger vs. SM1 tagger):
    • SM2 is better under B3 sys (F 73.9 vs. 70.8) and under CEAF sys (66.3 vs. 63.7).
    • B3 0 reverses this ranking (68.4 vs. 70.1).
    • CEAF orig gives very low, unstable scores (58.2 vs. 48.5).
  • Code availability: implementations of both metrics, with extended examples, are released open source at h-its.org/nlp/download.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Part-of-speech tagging

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cai2010evaluation,
  title = {{Evaluation Metrics For End-to-End Coreference Resolution Systems}},
  author = {Cai, Jie and Strube, Michael},
  booktitle = {Proceedings of the SIGDIAL 2010 Conference},
  year = {2010},
  url = {https://aclanthology.org/W10-4305/}
}

References

22 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bagga, Amit & Breck Baldwin (1998). Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pp. 563–566. find our card
  2. Bengtson, Eric & Dan Roth (2008). Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25-27 October 2008, pp. 294–303. find our card
  3. Culotta, Aron, Michael Wick & Andrew McCallum (2007). First-order probabilistic models for coreference resolution. In Proceedings of Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, Rochester, N.Y., 22–27 April 2007, pp.… find our card
  4. Denis, Pascal & Jason Baldridge (2007). Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, Rochester, N.Y., 22–27… find our card
  5. Haghighi, Aria & Dan Klein (2007). Unsupervised coreference resolution in a nonparametric Bayesian model. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic, 23–30 June 2007, pp. 848–855. find our card
  6. Luo, Xiaoqiang (2005). On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005, pp. 25–32. find our card
  7. Luo, Xiaoqiang, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla & Salim Roukos (2004). A mention-synchronous coreference resolution algorithm based on the Bell Tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, 21–26 July 2004, pp. 136–… find our card
  8. Mitchell, Alexis, Stephanie Strassel, Shudong Huang & Ramez Zakhary (2004). find
  9. ACE 2004 Multilingual Training B 0 3 Pr 91.1 91.3 3 find
  10. B all Pr 85.8 87.1 3 B r&n Pr 78.7 84.9 F 70.1 68.4 R 65.1 64.3 F 74.0 73.9 R 65.1 64.3 F 71.2 73.2 CEAF r&n Pr 59.8 66.2 CEAF orig Pr 39.9 56.6 R 62.1 60.0 R 62.0 60.0 F 48.5 58.2 F 60.9 62.9 Corpus. LDC2005T09, Philadelphia, Penn.: Linguistic Data Consortium. find
  11. Nicolae, Cristina & Gabriel Nicolae (2006). BestCut: A graph algorithm for coreference resolution. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Australia, 22–23 July 2006, pp. 275–283. find our card
  12. Ponzetto, Simone Paolo & Michael Strube (2006). Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, New York, N.Y., 4–9 June 200… find our card
  13. Rahman, Altaf & Vincent Ng (2009). Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Singapore, 6-7 August 2009, pp. 968–977. find our card
  14. Rand, William R. (1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336):846–850. find
  15. Recasens, Marta & Eduard Hovy (2010). BLANC: Implementing the Rand index for coreference evaluation. Submitted. find
  16. Soon, Wee Meng, Hwee Tou Ng & Daniel Chung Yong Lim (2001). A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. find our card
  17. Stoyanov, Veselin, Nathan Gilbert, Claire Cardie & Ellen Riloff (2009). Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International… find our card
  18. Versley, Yannick, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang & Alessandro Moschitti (2008). BART: A modular toolkit for coreference resolution. In Companion Volume to the Proceedings of the 46th Annual Meeting of the Association for Computational… find our card
  19. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly & Lynette Hirschman (1995). A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pp. 45–52. San Mateo, Cal.: Morgan Kaufmann. find our card
  20. Witten, Ian H. & Eibe Frank (2005). Data Mining: Practical Machine Learning Tools and Techniques (2nd ed.). San Francisco, Cal.: Morgan Kaufmann. find
  21. Yang, Xiaofeng, Jian Su & Chew Lim Tan (2008). A twin-candidate model for learning-based anaphora resolution. Computational Linguistics, 34(3):327–356. find our card
  22. A B 3 sys Example Output find

↑ back to top

This page in Markdown →