Elamur Research⌕
☰
home / papers / 2024 / A Controlled Reevaluation of Coreference Resolution Models

A Controlled Reevaluation of Coreference Resolution Models

Summary

Every leading coreference resolution (CR) system finetunes a pretrained language model. Because each paper uses a different backbone and training setup, it is unclear whether reported gains come from new task-specific architectures or simply from stronger language models. The authors reimplemented five CR models (C2F, S2E, WLC, LingMess, and the decoder-based Link-Append) in Hugging Face Transformers and confirmed that they reproduce the published scores. They then retrained the encoder-based models on a shared DeBERTa backbone with a common hyperparameter search space, and compared encoder and decoder systems at matched model sizes. Evaluation covered OntoNotes (CoNLL-2012), OntoGUM for out-of-domain genres, and GAP for pronoun resolution.

At comparable size, encoder-based models beat decoder-based ones in accuracy, speed and memory. Under controlled conditions, the oldest model, C2F, is competitive and transfers best to new genres. Most of the past five years of F1 improvement disappears once the language model is held fixed. The authors argue that future CR work should separate architectural effects from backbone effects.

Key points

  • Reimplementations match the published OntoNotes test scores closely. For example, C2F with BERT-large reaches 77.1 against the reported 76.9, and Link-Append with mT5-XXL reaches 83.3. Code is released at github.com/ianporada/coref-reeval.
  • Controlled factors:
    • the DeBERTa encoder, at base (138M) and large (405M) sizes;
    • training epochs, searched over {25, 50, 125};
    • task-head ffnn_size, searched over {1024, 2048, 3072, 4096};
    • full-length document input for C2F.
  • LingMess scaled to 1.5B parameters scores 82.5 CoNLL F1 on the CoNLL-2012 test set. This equals the decoder-based ASP model at 11B parameters.
  • With DeBERTa-large on the dev sets, C2F has the best OntoGUM score (66.9). It is, however, the slowest and most memory-hungry model: 8.7 GB and 129.3 ms/doc on an A100.
  • Data used:
    • OntoNotes 5.0 with CoNLL-2012 v4 splits of 1940/343/348 document parts;
    • all 213 OntoGUM documents;
    • the 2000-example GAP validation set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2404.00727

@inproceedings{porada2024controlled,
  title = {{A Controlled Reevaluation of Coreference Resolution Models}},
  author = {Porada, Ian and Zou, Xiyuan and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/3bnksph2i9te},
  eprint = {2404.00727},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/3bnksph2i9te}
}

References

33 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations. find our card
  2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. find
  3. Jerry R Hobbs. 1978. Resolving pronoun references. Lingua, 44(4):311–338. language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 993–1005, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find our card
  4. Jerry R Hobbs. 1979. Coherence and coreference. Cognitive science, 3(1):67–90. find
  5. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, New York City, USA. Association for Computational Linguistics. find
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pre-training approach. arXiv:1907.11692. find
  7. Jing Lu and Vincent Ng. 2020. Conundrums in entity coreference resolution: Making sense of the state of the art. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6620–6631, Online. Association for Computational Linguistics. find our card
  8. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… find our card
  9. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find our card
  10. Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coreference resolution without span representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 14–… find our card
  11. Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations. find
  12. Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Pra-sojo, Phil Blunsom, and Adhiguna Kuncoro. 2023. On “scientific debt” in NLP: A case for more rigour in language model pre-training research. In Proceedings of the 61st Annual Meeting of the Association for Computa… find
  13. Tuan Manh Lai, Trung Bui, and Doo Soon Kim. 2022. End-to-end neural coreference resolution revisited: A simple yet effective baseline. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8147–8151. IEEE. find
  14. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  15. Shon Otmazgin, Arie Cattan, and Yoav Goldberg. 2023. LingMess: Linguistically informed multi expert scorers for coreference resolution. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2752–2760, Dubrovnik, Croatia. Association for… find our card
  16. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–… find our card
  17. Massimo Poesio, Juntao Yu, Silviu Paun, Abdulrahman Aloraini, Pengcheng Lu, Janosch Haber, and Derya Cokal. 2023. Computational models of anaphora. Annual Review of Linguistics, 9:561–587. find
  18. Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Investigating failures to Cheung. 2023. generalize for coreference resolution models. arXiv:2303.09092. find our card
  19. Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autoregressive structured prediction with Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in… find
  20. Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
  21. Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533, Online. Association for Computational Linguistics. find our card
  22. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language… find our card
  23. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  24. Shubham Toshniwal, Haoyue Shi, Bowen Shi, Lingyu Gao, Karen Livescu, and Kevin Gimpel. 2020. A cross-task analysis of text span representations. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 166–176, Online. Association for Computational Linguistics. find
  25. Zdeněk Žabokrtský, Miloslav Konopík, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, Daniel Zeman, and Yilun Zhu. 2022. Findings of the shared task on multilingual coreference resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Corefere… find
  26. Shubham Toshniwal, Patrick Xia, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2021. On generalization in coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 111–120, Punta Cana, Dominican Republic. Association for Comput… find our card
  27. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581– 612. find
  28. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find our card
  29. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021a. Anatomy of OntoGUM—Adapting GUM to the OntoNotes scheme to evaluate robustness of SOTA coreference algorithms. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 141–149, Punta Cana, Dominican… find our card
  30. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. find our card
  31. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021b. OntoGUM: Evaluating contextualized SOTA coreference resolution on 12 more genres. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (… find our card
  32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find our card
  33. Language Resource References Weischedel, Ralph and others. 2013. OntoNotes Release 5.0. Defense Advanced Research Projects Agency, GALE Program. Linguistic Data Consortium, LDC2013T19, ISLRN 151-738-649-048-2. [link]. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/porada-2024-controlled-reevaluation-coreference-resolution-models/