Elamur Research⌕
☰
home / papers / 2013 / Critical Reflections on Evaluation Practices in Coreference Resolution

Critical Reflections on Evaluation Practices in Coreference Resolution

Summary

This paper asks how well the standard coreference resolution (CR) metrics match human judgement of CR quality. The author argues that discussion of these metrics typically relies on idealized entities and imagined mistakes, so she examines real texts and real system errors instead. First, she computes Kendall τ rank correlations between the metrics' rankings of systems in the CoNLL 2011 and CoNLL 2012 shared tasks. The metrics often disagree, and their agreement shifts noticeably between the two years.

Second, she manually analyses key/response entity pairs from five short OntoNotes-based texts, with system output from Stanford's Multi-Pass Sieve resolver. She sorts the pairs into four intuitive categories and then formalizes these categories with set-logic conditions and an overlap relation. The analysis shows that humans judge success by how much informative content survives in the response. Current metrics largely ignore this, and strict mention-boundary scoring can make it worse. The paper proposes adding mention information weight to improve both CR and its evaluation.

Key points

  • Metrics studied: MUC, B3, CEAFe, CEAFm and BLANC, plus MELA, the official CoNLL score (a weighted average of MUC, B3 and CEAFe).
  • CoNLL 2011 correlations:
    • Kendall τ fell as low as 0.22, between CEAFe and MUC.
    • The two link-based metrics, MUC and BLANC, reached only 0.35.
    • B3 and CEAFm agreed most among non-composite metrics, at 0.56.
  • CoNLL 2012 correlations: values were higher overall, ranging from 0.46 to 0.70.
  • Error-analysis data:
    • Five CoNLL 2011 development texts, one each from the BC, BN, MZ, NW and WB genres, with gold annotation from OntoNotes 4.0.
    • 64 mentions in 21 key entities.
    • Only perfect matches (19% of entities, 14% of mentions) received unanimous scores across metrics.
  • Effect of strict boundaries: under exact-boundary matching, entity NW1 averaged an F-score of 56.67%. Entities BC45 and BN1 scored 88.32% and 75.68%, even though all three lost comparable information.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{holen2013critical,
  title = {{Critical Reflections on Evaluation Practices in Coreference Resolution}},
  author = {Holen, Gordana Ilić},
  booktitle = {Proceedings of the 2013 NAACL HLT Student Research Workshop},
  year = {2013},
  url = {https://aclanthology.org/N13-2001/}
}

References

12 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, pages 563 – 566, Granada, Spain. find our card
  2. Pascal Denis and Jason Baldridge. 2009. Global joint models for coreference resolution and named entity classication. Procesamiento del Lenguaje Natural, 42:87–96. find
  3. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 Shared Task. In Proceedings of the 15th Conference on Computational Natural Language Learning: Shared Task, pages 28–34, P… find our card
  4. Xiaoqiang Luo, Abe Ittycheriah Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 136–143, Barcelona, Spain. find our card
  5. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 25–32, Vancouver, Canada. find our card
  6. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the 15th Conference on Computational Natural Language Learning: Shared Task, pages 1–27, Portland, Oregon. find our card
  7. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Joint Conference on EMNLP and CoNLL: Shared Task, pages 1–40, Jeju Island, Korea. find our card
  8. W. M. Rand. 1971. Objective criteria for evaluation of clustering methods. Journal of American Statistical Association, 66(336):846–850. find
  9. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering, 17(04):485–510. find
  10. Marta Recasens, Lluis Màrquez, Emili Sapena, M. Antònia Marti, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 1–… find our card
  11. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message understanding Conference (MUC-6), pages 45–52, San Francisco, CA. find our card
  12. Xiaofeng Yang, Guodong Zhou, Jian Su, and Chew Lim Tan. 2003. Coreference resolution using competition learning approach. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 176–183, Sapporo, Japan. find our card

↑ back to top

This page in Markdown →