Elamur Research
☰
home / papers / 2023 / Coreference Resolution through a seq2seq Transition-Based System

Coreference Resolution through a seq2seq Transition-Based System

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.tacl-1.13.pdf

Summary

The paper reframes coreference resolution, which is usually handled by encoder-only models (e.g., SpanBERT) with span search and separate mention-detection stages, as a pure text-to-text task. The authors build transition systems on top of multilingual T5 (mT5). The document is processed one sentence at a time. The input string holds earlier sentences annotated with the clusters found so far, followed by the current sentence. The model then outputs a series of actions, ending with SHIFT. The best variant, Link-Append, either attaches a mention to an existing cluster or links it to a previous mention to start a new cluster. It uses plain greedy decoding, with no dedicated mention detector and no higher-order inference. Trained only on CoNLL-2012 data, it sets new state-of-the-art scores for English, Arabic, and Chinese. On SemEval-2010, it also performs strongly in zero-shot, few-shot, and fully supervised settings. This shows that general-purpose generative models can beat specialized coreference architectures, and the released models offer a starting point for transfer to new languages.

Key points

  • CoNLL-2012 test F1 scores:
    • English: 83.3, which is +2.3 over Dobrovolskii (2021).
    • Arabic: 68.5 (+4.1).
    • Chinese: 74.3 (+5.3).
  • The backbone is xxl-mT5. Ablations show the importance of key design choices:
    • Removing cluster annotations from the input (the Link-only system) costs 1.8 F1.
    • Swapping in English-only xxl-T5.1.1 costs 0.5 F1.
    • Mention-Link-Append reaches 82.6 dev F1, 0.6 below Link-Append.
  • On SemEval-2010, zero-shot transfer from an English-only model beats earlier approaches for 3 of 4 languages. Supervised results exceed the previous state of the art for all five tested languages.
  • Error analysis covers 14.5k development-set predictions:
    • Hallucinated mentions occur in only 11 cases (0.07%).
    • Ambiguous mention matches occur in 84 cases (0.6%).
  • Published in TACL vol. 11 (2023). Code and models are released at google-research/coref_mt5.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Dependency parsing

Cited by

44 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 43 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00543 · arXiv 2211.12142

@article{bohnet2023coreference,
  title = {{Coreference Resolution through a seq2seq Transition-Based System}},
  author = {Bohnet, Bernd and Alberti, Chris and Collins, Michael},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00543},
  eprint = {2211.12142},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00543}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Conference on Empirical Methods in Natural Language Processing, pages 7670–7675, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.605 ↗ our card
  2. Abdulrahman Aloraini, Juntao Yu, and Massimo Poesio. 2020. Neural coreference resolution for Arabic. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, pages 99–110, Barcelona, Spain (online). Association for Computational Linguistics. find
  3. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021. Glam: Efficient scaling of language models with mixture-of-experts. CoRR, abs/2112.06905. ↗ our card
  4. Giuseppe Attardi, Maria Simi, and Stefano Dei Rossi. 2010. TANL-1: Coreference resolution by parse analysis and similarity clustering. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 108–111, Uppsala, Sweden. Association for Computational Linguistics. find
  5. Semere Kiros Bitew, Johannes Deleu, Chris Develder, and Thomas Demeester. 2021. Lazy low-resource coreference resolution: A study on leveraging black-box translation tools. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 57–62, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10 .1162/neco.1997.9.8.1735, PubMed: 9377276 ↗
  7. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10 .1162/tacl_a_00300 ↗ our card
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, GirishSastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165. ↗
  9. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. https:// doi.org/10.18653/v1/D19-1588 ↗ our card
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  11. Daniel Jurafsky and James H. Martin. 2021. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, third edition. find
  12. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. find
  13. Vladimir Dobrovolskii. 2021. Word-level coreference resolution. In Proceedings of the 2021 find our card
  14. Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coreference resolution without span representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 14–19, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-short.3 doi.org/10.1162/coli.07-056-R1 -07-027 ↗ our card
  15. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1202 ↗
  16. Hamidreza Kobdani and Hinrich Schütze. 2010. SUCRE: A modular system for coreference resolution. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 92–95, Uppsala, Sweden. Association for Computational Linguistics. find
  17. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL -Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
  18. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. ↗
  20. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  21. Marta Recasens, Lluı́s Màrquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 1–8, Uppsala, Sweden. Association for Computational Linguistics. https://doi.org/10.3115/1621969.1621982 ↗
  22. Bonan Min. 2021. Exploring pre-trained transformers and bilingual transfer learning for Arabic coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 94–99, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.crac-1.10 ↗
  23. Ina Roesiger and Jonas Kuhn. 2016. IMS Hot-Coref DE: A data-driven co-reference resolver for German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 155–160, Portorož, Slovenia. European Language Resources Association (ELRA). find
  24. Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proceedings of the Eighth International Conference on Parsing Technologies, pages 149–160, Nancy, France. find our card
  25. Fynn Schröder, Hans Ole Hatzel, and Chris Biemann. 2021. Neural end-to-end coreference resolution for German in different domains. In Proceedings of the 17th Conference on Natural Language Processing (KONVENS Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513–553. https://2021), pages 170–181, Düsseldorf, Germany. KONVENS 2021 Organizers. Computational Linguistics. https://doi .org/10.18653/v1/2020.acl-main .622 ↗ our card
  26. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog applications. CoRR, abs/2201.08239. ↗ our card
  27. Patrick Xia and Benjamin Durme. 2021. Moving on from OntoNotes: Coreference resolution model transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5241–5256, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  28. Patrick Xia, João Sedoc, and Benjamin Van Durme. 2020. Incremental neural coreference resolution in constant memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8617–8624. Association for Computational Linguistics, Online. find our card
  29. Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533, Online. Association for Computational Linguistics. ↗ our card
  30. Kellie Webster and James R. Curran. 2014. Limited memory incremental coreference resolution. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2129–2139, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. find
  31. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  32. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Juntao Yu, Alexandra Uma, and Massimo Poesio. 2020. A cluster ranking model for full anaphora resolution. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 11–20, Marseille, France. European Language Resources Association. find our card