Elamur Research
☰
home / papers / 2017 / Lexical Features in Coreference Resolution: To be Used With Caution

Lexical Features in Coreference Resolution: To be Used With Caution

What it is about

Learned coreference resolvers owe much of their gain over rule systems to lexical features (surface words or embeddings of heads, first/last words, context). The paper shows the cost. On WikiCoref, annotated with the same guidelines as CoNLL-2012, the Stanford rule-based sieve system (CoNLL F1 about 51.0) is on par with or ahead of berkeley, cort and deep-coref (51.0 down to 49.9, 52.7 for the best deep-coref variant), although on the CoNLL test set the learned systems are 6–10 points ahead. The reason is overlap: a large share of the coreferent mention-head pairs in the CoNLL test set already occur, as coreferent pairs, in training (38–51% by genre, 76–77% for the pivot/Bible genre), so lexical features memorise rather than generalise. The authors call for out-of-domain evaluation and features that capture linguistic properties rather than words.

Ideas

For Mova

languages/en/35-reference-and-coreference.md §8–9. Code: coref/src/eval.rs (per-genre and overlap reports), coref/src/sieve.rs. Measures: CoNLL F1 per GUM genre on dev; head-pair overlap between dev and test.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-2003.pdf

Summary

Modern coreference resolvers depend heavily on lexical information, either surface forms or word embeddings, to represent mentions and their contexts. Moosavi and Strube ask whether this knowledge transfers beyond the training data. They compare a rule-based system, berkeley, cort and deep-coref on the CoNLL test set and on WikiCoref, which follows the same annotation guidelines. They also measure how many test mentions already appear as coreferent in the CoNLL training data. Finally, they run in-domain and out-of-domain experiments on two genres: pt, with high overlap, and wb, with low overlap.

The large lead of lexicalized systems over the rule-based baseline mostly disappears on WikiCoref. These systems, and the neural deep-coref in particular, gain most when the training and test data share mentions. deep-coref's links also favour head pairs it saw during training.

The authors argue that evaluating only on the official CoNLL split rewards exploiting the dataset rather than solving coreference. They call for out-of-domain evaluation as standard practice, using held-out CoNLL genres or corpora such as WikiCoref, MUC or ACE.

Key points

  • On the CoNLL test set, deep-coref beats the Stanford rule-based system by about 10 CoNLL points (roughly 65.6 vs. 55.6). On WikiCoref the gap appears to shrink to under 2 points (about 53.1 vs. 51.8), with cort falling below the rule-based baseline.
  • Across the CoNLL test genres, 37% to 77% of the coreferent mentions that are not pronouns had already appeared as coreferent in the training data. The pt (Bible) genre has the most overlap (77%) and tc the least.
  • Adding the pt genre to training raises deep-coref's LEA score by more than 13%. For the low-overlap wb genre, the gains are much smaller, and some systems even do better when wb is left out.
  • deep-coref links 49 nominal pairs without head match on the development set. Only 5 of these pairs are unseen in training, and all 5 are wrong.
  • The authors suggest pruning rare lexical features and using embeddings mainly for context rather than for the mentions themselves. They recommend selecting models with LEA, which consistently outperformed selection by CoNLL score.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2003 · arXiv 1704.06779

@inproceedings{moosavi2017lexical,
  title = {{Lexical Features in Coreference Resolution: To be Used With Caution}},
  author = {Moosavi, Nafise Sadat and Strube, Michael},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2003},
  eprint = {1704.06779},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-2003}
}

References

15 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005. pages 25–32. find
  2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998. pages 563–566. find our card
  3. Sebastian Martschat and Michael Strube. 2014. Recall error analysis for coreference resolution. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014. pages 2070–2081. http://www.aclweb.org/anthology/D14-1221.pdf. ↗ our card
  4. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Tex., 1–5 November 2016. pages 2256–2262. https://www.aclweb.org/anthology/D16-1245.pdf. ↗ our card
  5. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics 3:405–418. http://www.aclweb.org/anthology/Q15-1029.pdf. ↗ our card
  6. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. http://www.aclweb.org/anthology/P16-1061.pdf. ↗ our card
  7. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. pages 632–642. http://www.aclweb.org/anthology/P16-1060.pdf. ↗ our card
  8. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013. pages 1971–1982. http://www.aclweb.org/anthology/D13-1203.pdf. ↗ our card
  9. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Md., 22–27 June 2014. pages 30–35. http://www.aclweb.org/anthology/P14-2006.pdf. ↗ our card
  10. Abbas Ghaddar and Philippe Langlais. 2016a. Coreference in Wikipedia: Main concept resolution. In Proceedings of the 20th Conference on Computational Natural Language Learning, Berlin, Germany, 11–12 August 2016. pages 229–238. https://www.aclweb.org/anthology/K16-1023.pdf. ↗
  11. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6). Morgan Kaufmann, San Mateo, Cal., pages 45–52. find our card
  12. Abbas Ghaddar and Philippe Langlais. 2016b. WikiCoref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the 10th International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016. http://www.lrec-conf.org/proceedings/lrec2016/pdf/192 Paper.pdf. ↗ our card
  13. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics 39(4):885–916. https://www.aclweb.org/anthology/J13-4004.pdf. ↗ our card
  14. Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Col., 31 May – 5 June 2015. pages 970–976. http://www.aclweb.org/anthology/N15-1098. ↗ our card
  15. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 find our card