Elamur Research
☰
home / papers / 2019 / Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection

Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection

What it is about

Scoring coreference by maximum spans mixes coreference errors with boundary errors such as PP attachment: «an extensive presence» and «an extensive presence, of course in this country» count as different mentions, and the system is penalised as if it had linked the wrong thing. MUC, ACE and ARRAU annotate minimum spans by hand, CoNLL-2012 does not. MINA extracts a minimum span automatically from one constituency tree: walk the mention's subtree breadth-first through NP-internal nodes and keep the shallowest terminal nodes that contain a word other than a determiner or conjunction (so a name keeps all its words, unlike a single head word, which makes «Mary Green» and «John Green» both «Green»). Gold and system mentions are reduced on the same tree and compared by their minimum spans. MINA spans agree with expert minimum spans better than head words, and minimum-span scores reduce the effect of parser differences on system rankings.

Ideas

For Mova

languages/en/35-reference-and-coreference.md §7 (annotating coreference), §9.3. Code: coref/src/score.rs (matching mode), coref/src/sieve.rs, coref/src/mention.rs. Measures: CoNLL F1 on GUM dev in three matching modes, gold vs SLM trees; precision of name sieves.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1408.pdf

Summary

Coreference resolvers are usually scored by matching the full (maximum) extent of each mention. As a result, the scores also reflect parsing difficulties such as prepositional phrase attachment, along with small boundary slips like an extra comma or missing quotation marks. Corpora such as MUC, ACE and ARRAU address this by having experts mark a minimum span for each mention. That extra annotation is expensive, so large resources like CoNLL-2012 omit it.

The authors propose MINA, an algorithm that derives minimum spans automatically from constituency parse trees. It walks a mention's subtree breadth-first and ignores branches whose tags are not nominal (e.g., PP). It then keeps the shallowest terminal nodes that contain something other than a determiner or conjunction. Gold and system mentions are both reduced using the same tree, so a system span that is too long or too short can still match its gold counterpart.

On MUC and ARRAU, MINA spans agree with expert annotations better than Collins head words do. Scoring on minimum spans leads to fairer comparisons of resolvers, especially across datasets.

Key points

  • MINA treats NP, NML, QP and NX as acceptable tags for noun phrases and only VP for verb phrases. If no valid node is found, it falls back to the maximum span. Using gold parses, the CoNLL-2012 development set contains 14 such fallback cases, and ten of them consist of a single word.
  • The baseline for comparison is Collins' head finder rules, applied without the named-entity adjustment used by Peng et al. (2015). MINA spans match the manual minimum spans of MUC and ARRAU more closely.
  • Minimum-span evaluation shrinks the performance gap between gold and system mentions by roughly two points. The effect is strongest out of domain, where most mismatches come from:
    • appositives,
    • quotation marks,
    • trailing commas.
  • Under maximum spans, Peng et al. (2015) performs on par with cort. Under minimum spans, cort leads by about one point, suggesting that part of the original parity came from boundary detection. In out-of-domain tests, e2e-coref showed the smallest maximum-vs-minimum difference (1.4 points).
  • MINA is built into the CoVal scorer (github.com/ns-moosavi/coval), which reports standard metrics on both maximum and MINA spans. So far it has only been validated for English.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1408 · arXiv 1906.06703

@inproceedings{moosavi2019using,
  title = {{Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection}},
  author = {Moosavi, Nafise Sadat and Born, Leo and Poesio, Massimo and Strube, Michael},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1408},
  eprint = {1906.06703},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1408}
}

References

18 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pages 563–566. find our card
  2. Kevin Clark and Christopher D. Manning. 2016a. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Berlin, Germany. Association for Computational Linguistics. find our card
  3. Kevin Clark and Christopher D. Manning. 2016b. Deep reinforcement learning for mention-ranking coreference models. pages 2256–2262, Austin, Texas. Association for Computational Linguistics. find our card
  4. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia, Penn. find our card
  5. Abbas Ghaddar and Philippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the 10th International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016. find our card
  6. Lynette 1997. Hirschman and Nancy Chinchor. MUC-7 coreference task definition, http://www.muc.saic.com/proceedings/ . ↗
  7. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  8. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692. Association for Computational Linguistics. find our card
  9. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. ↗ our card
  10. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find our card
  11. Alexis Mitchell, Stephanie Strassel, Mark Przybocki, JK Davis, George Doddington, Ralph Grishman, Adam Meyers, Ada Brunstain, Lisa Ferro, and Beth Sundheim. 2002. ACE-2 Version 1.0. LDC2003T11, Philadelphia, Penn.: Linguistic Data Consortium. find
  12. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  13. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19, Vancouver, Canada. Association for Computational Linguistics. ↗ our card
  14. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan. Association for Computational Linguistics. find our card
  15. Nafise Sadat Moosavi and Michael Strube. 2018. Using linguistic features to improve the generalization capability of neural coreference resolvers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 193– 203, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  16. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
  17. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In Proceedings of the 19th Conference on Computational Natural Language Learning, Beijing, China, 30–31 July 2015, pages 12–21. find
  18. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card