Elamur Research⌕
☰
home / papers / 2023 / Are Large Language Models Robust Coreference Resolvers?

Are Large Language Models Robust Coreference Resolvers?

Summary

Most coreference systems for new domains or languages need annotated in-domain data, which is expensive to collect. Earlier tests of LLM coreference used small sentence-level puzzles such as the Winograd Schema Challenge. Le and Ritter test instruction-tuned LMs on realistic document-level benchmarks, mainly CoNLL-2012/OntoNotes 5.0. They give the model a document with candidate mentions marked by special tokens. The model then writes a cluster ID next to each mention. These mentions are either gold or predicted by Stanford's rule-based dcoref. The authors compare Llama-2-Chat, CodeLlama, InstructGPT, ChatGPT and GPT-4 with supervised and unsupervised baselines.

GPT-4 and InstructGPT beat unsupervised systems, and the margin grows when gold mentions are given. However, they remain well below supervised models and depend heavily on mention detection quality. The LMs also transfer reasonably well across domains, languages and time periods. Even so, continued fine-tuning is better when a few annotated target documents exist. The work shows that prompting is a practical fallback for low-resource coreference settings.

Key points

  • Prompt template: In a pilot study, a QA-style prompt reached about 61 F1 versus dcoref's 72. The document-level marking template reached about 81, so it was used for all later experiments.
  • OntoNotes test set, CoNLL F1, predicted mentions: GPT-4 scored 62.9 and InstructGPT 60.1. The baselines were weak-SpanBERT at 59.3 and dcoref at 58.6. The supervised systems were far ahead: coref-mt5 at 83.3 and SpanBERT+e2e at 79.6.
  • OntoNotes test set, gold mentions: GPT-4 reached 88.4, InstructGPT 80.8, and dcoref 72.9.
  • Llama 2 models: All of them trail dcoref. CodeLlama clearly beats Llama-2-Chat, and CodeLlama-7B is roughly on par with Llama-2-Chat-70B.
  • Main weakness: Mention detection is the bottleneck. InstructGPT's mention detection F1 is 46.5 versus 76.6 for dcoref, and its recall on nominal phrases is especially poor (18.7 vs 52.7). Pronoun resolution is its strongest category.
  • Time robustness: The authors built three silver-annotated test sets of 56 Wall Street Journal articles each: WSJ-1989, WSJ-2019 and WSJ-2023, labelled with SpanBERT. Prompted LMs varied less across periods than dcoref. Code and data are available at github.com/nle18/coref-llms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

26 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 19 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.14489 · arXiv 2305.14489

@misc{le2023are,
  title = {{Are Large Language Models Robust Coreference Resolvers?}},
  author = {Le, Nghia T. and Ritter, Alan},
  year = {2023},
  doi = {10.48550/arxiv.2305.14489},
  eprint = {2305.14489},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14489}
}

References

37 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics. find our card
  2. Oshin Agarwal and Ani Nenkova. 2022. Temporal effects on pre-trained models for language processing tasks. ↗
  3. Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. find
  4. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. In EMNLP. find
  5. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find our card
  6. Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022. Ask me anything: A simple strategy for prompting language models. arXiv:2210.02441. ↗
  7. Nghia T Le, Fan Bai, and Alan Ritter. 2022. Few-shot anaphora resolution in scientific protocols via mixtures of in-context experts. EMNLP Findings. find our card
  8. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. Prompting language models for linguistic structure. find
  9. Bernd Bohnet, Chris Alberti, and Michael Collins. 2022. Coreference resolution through a seq2seq transition-based system. arXiv preprint arXiv:2211.12142. ↗ our card
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. find
  11. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find our card
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find
  13. Nupoor Gandhi, Anjalie Field, and Emma Strubell. 2022. Mention annotations alone enable efficient domain adaptation for coreference resolution. ↗
  14. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  15. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics. ↗ our card
  16. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  17. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. find
  18. Shuheng Liu and Alan Ritter. 2023. Do conll-2003 named entity taggers still work well in 2023? ACL. ↗
  19. Tianyu Liu, Yuchen Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autoregressive structure prediction with language models. In Findings of the Association for Computational Linguistics: EMNL 2022. find
  20. Jing Lu and Vincent Ng. 2020. Conundrums in entity coreference resolution: Making sense of the state of the art. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  21. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences. ↗
  22. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find our card
  23. OpenAI. 2022. Introducing chatgpt. find
  24. OpenAI. 2023. Gpt-4 technical report. find our card
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ↗
  26. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task. find our card
  27. Marta Recasens, Lluís Màrquez, Emili Sapena, M. Antònia Martí, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation. find our card
  28. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code. ↗ our card
  29. Alessandro Stolfo, Chris Tanner, Vikram Gupta, and Mrinmaya Sachan. 2022. A simple unsupervised approach for coreference resolution using rule-based weak supervision. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics. find
  30. Shubham Toshniwal, Patrick Xia, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2021. On generalization in coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference. find our card
  31. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ↗
  32. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A Large Training Corpus for Enhanced Processing. find
  33. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  34. Zhaofeng Wu and Matt Gardner. 2021. Understanding mention detector-linker interaction in neural coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference. ↗ our card
  35. Patrick Xia and Benjamin Van Durme. 2021. Moving on from OntoNotes: Coreference resolution model transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. find
  36. Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Chris Tanner. 2022. What GPT knows about who is who. In Proceedings of the Third Workshop on Insights from Negative Results in NLP. find
  37. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32. the cluster information (We extract cluster information using a simple fuzzy string matching algorithm by comparing the output text to input text, sentence-by-sentence). A.2 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/le-2023-large-language-models-robust-coreference/