Elamur Research⌕
☰
home / papers / 2014 / Coreference Resolution Evaluation for Higher Level Applications

Coreference Resolution Evaluation for Higher Level Applications

Summary

Standard coreference metrics (MUC, BCUB, CEAF, BLANC) score system output as a clustering problem and ignore the linear order of mentions. This makes the scores hard to translate into benefit for downstream tasks. Tuggener proposes an evaluation framework built around three ways applications consume coreference:

  • Entity-distribution models such as entity grids or event sequences. These need a gapless chain, so a mention counts as correct only if its immediate antecedent matches the key.
  • Applications needing a local nominal antecedent such as summarization or MT gender agreement. Here the nearest preceding noun in the response must be a true antecedent.
  • Query-driven tasks such as sentiment analysis or biomedical relation mining. Response chains are aligned to key chains through an anchor (the first nominal mention), and the metric measures entity detection and entity mention quality.

Recall and Precision both include a "wrong linkage" error class, so they cover all key and system mentions. Scores can also be broken down by any mention feature, such as PoS or named entity class. Tested on three systems on the CoNLL 2012 English test set, the metrics give detailed diagnostics while broadly preserving the system ranking from the MELA score.

Key points

  • Recall is defined as tp/(tp+wl+fn) and Precision as tp/(tp+wl+fp). The wrong linkage (wl) concept comes from Durrett and Klein (2013). Singleton entities are excluded.
  • The evaluated systems were Berkeley (Durrett and Klein, 2013), IMS (Björkelund and Farkas, 2012) and Stanford (Lee et al., 2013). Their MELA scores were 61.62, 57.42 and 55.69 respectively.
  • Total F for immediate antecedents was 56.76 (Berkeley), 55.48 (IMS) and 52.19 (Stanford). For inferred nominal antecedents it was 55.37, 50.25 and 46.97.
  • Nouns score better under the nominal-antecedent criterion, while pronouns score better under the immediate-antecedent criterion. IMS beats Berkeley on pronouns for immediate antecedents. Its lower Recall comes mostly from false negatives, not wrong linkages.
  • In the anchor-based evaluation, overall Fφ was 63.41 (Berkeley), 55.24 (IMS) and 55.27 (Stanford). By entity class, IMS is weak on PERSON but best on GPE. Of the 4532 key chains, 476 (10.05%) contain no nominal mention. The scorers are freely available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/e14-4045

@inproceedings{tuggener2014coreference,
  title = {{Coreference Resolution Evaluation for Higher Level Applications}},
  author = {Tuggener, Don},
  booktitle = {Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers},
  year = {2014},
  doi = {10.3115/v1/e14-4045},
  url = {https://doi.org/10.3115/v1/e14-4045}
}

References

11 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Comput. Linguist., 34(1):1–34, March. find our card
  2. Anders Björkelund and Richárd Farkas. 2012. Data-driven multilingual coreference resolution using resolver stacking. In Joint Conference on EMNLP and CoNLL - Shared Task, CoNLL ’12, pages 49–55, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  3. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the 6th International Joint Conference on Natural Language Processing, pages 1366–1374. find our card
  4. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, October. Association for Computational Linguistics. find our card
  5. Micha Elsner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 125–129, Stroudsburg, PA, USA. Association… find
  6. Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: a framework for modeling the local coherence of discourse. Comput. Linguist., 21(2):203–225, June. find our card
  7. Joseph Irwin, Mamoru Komachi, and Yuji Matsumoto. 2011. Narrative schema as world knowledge for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CONLL Shared Task ’11, pages 86–92, Stroudsburg, PA, USA. Association for Comput… find our card
  8. June-Jei Kuo and Hsin-Hsi Chen. 2007. Cross-document event clustering using knowledge mining from co-reference chains. Inf. Process. Manage., 43(2):327–343, March. find
  9. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL… find our card
  10. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4). find our card
  11. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/tuggener-2014-coreference-resolution-evaluation-higher-level/