Elamur Research
☰
home / papers / 2019 / BERT for Coreference Resolution: Baselines and Analysis

BERT for Coreference Resolution: Baselines and Analysis

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1588.pdf

Summary

This paper tests how much a pretrained BERT encoder helps coreference resolution. It also looks at where the resulting system still fails. The authors start from c2f-coref (Lee et al., 2018), a span-ranking model, and swap its LSTM encoder, which used ELMo and GloVe inputs, for a fine-tuned BERT. Long documents have to be cut into segments. They try two ways of doing this: separate non-overlapping chunks ("independent"), and half-overlapping chunks whose token vectors are merged by a learned gate ("overlap"). Training uses English OntoNotes 5.0. The models are evaluated on OntoNotes and on the GAP pronoun dataset. BERT-large gives clear gains over the ELMo baseline on both, while BERT-base helps only slightly on OntoNotes. A manual error analysis shows that the large model is better at keeping similar but distinct entities apart. Long documents, pronouns in dialogue and paraphrased mentions remain hard, and longer context windows do not help. The study set a strong reference point for transformer-based coreference, and the code and models are public.

Key points

  • OntoNotes test (average F1 of MUC, B³, CEAF φ4): c2f-coref scores 73.0, BERT-base 73.9 (+0.9) and BERT-large 76.9 (+3.9). The overlap variant gives no gain over independent.
  • GAP (overall F1, trained only on OntoNotes): c2f-coref scores 73.5, BERT-base 82.8 and BERT-large 85.0 (+11.5).
  • Best segment length: 128 word pieces for BERT-base and 384 for BERT-large. Both get worse at 450 and 512 (dev F1 falls to 70.7 and 73.6), and F1 drops steadily as documents get longer.
  • Manual error count: on 15 OntoNotes dev documents, BERT-base made 93 cluster-level errors and BERT-large made 74. The main error types are pronouns (especially in conversations), mention paraphrasing and missing world knowledge.
  • Training setup: separate learning rates for BERT (1e-5) and the task layers (2e-4) added 2–3 points. Training BERT-large needs 32GB GPUs. The code is at github.com/mandarjoshi90/coref.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

176 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 156 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1588 · arXiv 1908.09091

@inproceedings{joshi2019bert,
  title = {{BERT for Coreference Resolution: Baselines and Analysis}},
  author = {Joshi, Mandar and Levy, Omer and Zettlemoyer, Luke and Weld, Daniel},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1588},
  eprint = {1908.09091},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1588}
}

References

22 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, We required a 32GB GPU to finetune BERT-large. pages 294–303, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  2. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. ↗
  3. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405–1415. Association for Computational Linguistics. find our card
  4. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262. Association for Computational Linguistics. find our card
  5. Pascal Denis and Jason Baldridge. 2008. Specialized models and ranking for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 660–669. Association for Computational Linguistics. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971–1982. Association for Computational Linguistics. find our card
  8. Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. 2019. End-to-end deep reinforcement learning based coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 660–665, Florence, Italy. Association for Computational Linguistics. find
  9. Eraldo Fernandes, Cı́cero dos Santos, and Ruy Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 41–48. Association for Computational Linguistics. find our card
  10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. ↗ our card
  11. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy. Association for Computational Linguistics. ↗
  12. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  13. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. find our card
  14. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1416–1426. Association for Computational Linguistics. find our card
  15. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 994–1004. Association for Computational Linguistics. find our card
  16. Fei Liu, Luke Zettlemoyer, and Jacob Eisenstein. 2019. The referential reader: A recurrent entity network for anaphora resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5918–5925, Florence, Italy. find
  17. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find our card
  18. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6297–6308. find our card
  19. Vincent Ng and Claire Cardie. 2002. Identifying anaphoric and non-anaphoric noun phrases to improve coreference resolution. In Proceedings of the 19th International Conference on Computational Linguistics - Volume 1, COLING ’02, pages 1–7, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  20. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227– 2237. Association for Computational Linguistics. find
  21. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40. Association for Computational Linguistics. find our card
  22. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced find