Elamur Research⌕
☰
home / papers / 2014 / An Extension of BLANC to System Mentions

An Extension of BLANC to System Mentions

Paper · 2014 · Xiaoqiang Luo, Sameer Pradhan, Marta Recasens et al.

Summary

BLANC is a link-based metric for evaluating coreference resolution. It scores coreference links and non-coreference links separately and averages the two F-measures. In its original form (BLANC-gold, Recasens and Hovy, 2011), it assumes that system mentions match gold mentions exactly. Real systems must detect mentions themselves, so they miss some gold mentions and add spurious ones. This makes the original metric unusable when gold mentions are unavailable, or when one score should cover both mention detection and coreference.

The authors restate BLANC-gold in set notation. They then show that the gold-mention requirement can be dropped by counting links that appear on only one side (key or response) as errors. Recall and precision reduce to correct links divided by all key or all response links. They prove that the extended metric equals BLANC-gold whenever the two mention sets coincide. They also define special handling for degenerate documents.

They added the metric to the public CoNLL scorer and scored the CoNLL-2011 and CoNLL-2012 official-track systems. The new BLANC correlates strongly with MUC, B-cubed and CEAF, which supports its use on predicted mentions.

Key points

  • Unified formulas. R_c = |C_k∩C_r|/|C_k| and P_c = |C_k∩C_r|/|C_r|, with analogous formulas for non-coreference links (N_k, N_r). BLANC is the arithmetic mean of F_c and F_n.
  • Boundary cases. Three are handled explicitly:
    • no links at all: score is 1 if the mention sets match, otherwise 0;
    • only singleton entities: BLANC = F_n;
    • one single entity: BLANC = F_c.
  • Toy example. Key {abc}{d} and response {bc}{de} give F_c = 2/5 and F_n = 4/7, so BLANC = 17/35.
  • Evaluation. The extended metric was implemented in the CoNLL reference coreference scorer. It was used to score CoNLL-2011 (English) and CoNLL-2012 (Arabic, Chinese, English) systems on predicted mentions.
  • Correlations. Pearson's r with MUC, B-cubed, CEAF-m and CEAF-e ranged from about 0.78 to 0.98 (p < 0.001), with the closest agreement for B-cubed and CEAF-m. OntoNotes does not annotate singletons, so the authors expect lower correlations on data that includes them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Harvard UniversityCambridge · United States112 works · 42,023 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Coreference resolution · 574 works

Cite

Cite as Xiaoqiang Luo, Sameer Pradhan, Marta Recasens and Eduard Hovy. 2014. An Extension of BLANC to System Mentions. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2005

@inproceedings{luo2014extension,
  title = {{An Extension of BLANC to System Mentions}},
  author = {Xiaoqiang Luo, Sameer Pradhan, Marta Recasens and Eduard Hovy},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2005},
  url = {https://doi.org/10.3115/v1/p14-2005}
}

References

9 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation (LREC’98), pages 563–566.
  2. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, New York City, USA, June. Association for Computational Linguisti… find
  3. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proc. of Human Language Technology (HLT)/Empirical Methods in Natural Language Processing (EMNLP).
  4. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1– 27, Portland, O…
  5. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea, July. Association for Computational Lin…
  6. W. M. Rand. 1971. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336):846–850. find
  7. M. Recasens and E. Hovy. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering, 17:485–510, 10. find
  8. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language…
  9. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, , and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In In Proc. of MUC6, pages 45–52.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2014-extension-blanc-system-mentions/