Elamur Research⌕
☰
home / papers / 2013 / Linguistically Aware Coreference Evaluation Metrics

Linguistically Aware Coreference Evaluation Metrics

Summary

Standard coreference scorers (MUC, B3, CEAF) treat mentions as interchangeable items, so any clustering task could be scored with them. Chen and Ng argue this hides real quality differences. A system that links a pronoun to a name helps a reader identify the entity. A system that only links two pronouns to each other does not, yet both receive equal credit. The authors show that MUC, B3, CEAF_m and CEAF_e can all be written in terms of three shared weight functions: for the common subset, the key chain and the system chain. Making only these functions sensitive to mention type yields linguistically aware variants of every metric at once: LMUC, LB3, LCEAF_m and LCEAF_e. Links are weighted by the most informative mention they involve (name > nominal > pronoun), and singleton chains get a separate weight. On an illustrative document with five system outputs, scored against gold mentions, the new metrics separate outputs that the original metrics score identically. They also punish pronoun-only linking. This gives the community a single, principled way to reward resolvers that actually support text understanding.

Key points

  • The framework uses four tunable weights: w_nam, w_nom and w_pro for links, plus w_sing for singleton chains. Each link receives the weight of the first matching rule: name, then nominal, then pronoun. Several weight settings (W1–W5) are compared.
  • Response (a) links only pronouns. Under the weight setting W4 its F-score falls from 73.7 to 54.4 (MUC→LMUC) and from 78.3 to 50.2 (B3→LB3).
  • The original metrics give responses (b), (c) and (d) identical scores (MUC F = 80.0). LMUC ranks them 58.5 < 66.2 < 73.0, matching how informative their antecedents are (pronoun, nominal, name).
  • w_sing does not affect LB3 or LCEAF_e. LMUC and LCEAF_m weigh singletons more as w_sing grows. Larger gaps between the type weights widen score differences between more and less informative resolvers.
  • Experiments use gold mentions. The authors note the metrics could be paired with Cai and Strube's (2010) method to handle system mentions. Their implementation was released at hlt.utdallas.edu.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chen2013linguistically,
  title = {{Linguistically Aware Coreference Evaluation Metrics}},
  author = {Chen, Chen and Ng, Vincent},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1193/}
}

References

8 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, page 563–566. find our card
  2. Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of the 11th Annual SIGDIAL meeting on Discourse and Dialogue, pages 28–36. find our card
  3. Harold W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83–97. find
  4. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32. find our card
  5. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1– 27. find our card
  6. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Languag… find our card
  7. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference resolution. Natural Language Engineering, 17(4):485– 510. find
  8. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference, pages 45–52. find our card

This page in Markdown → https://md.research.elamur.ai/free/papers/chen-2013-linguistically-aware-coreference-evaluation-metrics/