Elamur Research⌕
☰
home / papers / 2021 / Scaling Within Document Coreference to Long Texts

Scaling Within Document Coreference to Long Texts

Summary

Neural end-to-end coreference systems in the tradition of Lee et al. (2017) consume a lot of memory and run slowly on long inputs such as books. They have two main bottlenecks. They build representations for every candidate span and span pair, and each mention must be compared with all earlier candidates. The authors approximate this architecture at the token level. They score mention boundaries and antecedent links with bilinear functions over encoder token embeddings rather than span vectors. To limit antecedent comparisons, they split documents into token windows, keep only k-nearest-neighbour token links, use low-rank factorisation of the token similarity matrix, and discard non-essential candidate tokens from memory. The model was tested on CoNLL-2012, Litbank and MedMentions. Its F1 is close to strong span-based baselines, while training and inference are faster and lighter. This makes neural coreference practical for very long documents on modest hardware. That matters for corpora of novels, blogs or other lengthy texts.

Key points

  • Evaluation datasets were CoNLL-2012 (OntoNotes), Litbank (literary texts that require singleton mentions) and MedMentions (biomedical texts).
  • The system processed a novel of about two million tokens in roughly 14 minutes using 12GB of GPU RAM. Earlier end-to-end systems reached only about 11k tokens even with up to 48GB.
  • On Litbank it ran up to 10x faster and used less memory than the memory-based incremental model of Toshniwal et al. (2020b).
  • In the ablation table, the full model reaches F1 of about 78.0 (CoNLL), 75.9 (Litbank) and 61.8 (MedMentions). On Litbank, removing the subtoken strategy and mention (pre)training lowers F1 to about 68.2. The CoNLL state of the art remains Wu et al. (2020), at about 83 F1, but that approach is too costly for long texts.
  • Code and trained models are released at github.com/raghavlite/Scalable-Coreference. The token-level bilinear scoring idea was developed concurrently by Kirstain et al. (2021).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.343

@inproceedings{thirukovalluru2021scaling,
  title = {{Scaling Within Document Coreference to Long Texts}},
  author = {Thirukovalluru, Raghuveer and Monath, Nicholas and Shridhar, Kumar and Zaheer, Manzil and Sachan, Mrinmaya and McCallum, Andrew},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.343},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.343}
}

References

16 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green ai. Commun. ACM, 63(12):54–63. find our card
  2. David Bamman, Olivia Lewke, and Anya Mansoor. 2019. An annotated dataset of coreference in english literature. arXiv preprint arXiv:1912.01140. find our card
  3. Madeleine Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  4. Emma Strubell, Ananya Ganesh, and Andrew Mc-Callum. 2019. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243. find our card
  5. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel S Weld. 2019. Bert for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN… find our card
  6. Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coreference resolution without span representations. arXiv preprint arXiv:2101.00434. find our card
  7. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  8. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–… find our card
  9. Jing Lu and Vincent Ng. 2020. Conundrums in entity coreference resolution: Making sense of the state of the art. In Proceedings of the 2020 Conference de Scudery. 1601. find our card
  10. Artamenes. https://quod.lib.umich.edu/e/eebo/ A70988.0001.001 . ↗
  11. Shubham Toshniwal, Allyson Ettinger, Kevin Gimpel, and Karen Livescu. 2020a. Petra: A sparsely supervised memory model for people tracking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5415–5428. Association for… find
  12. Shubham Toshniwal, Sam Wiseman, Allyson Ettinger, Karen Livescu, and Kevin Gimpel. 2020b. Learning to ignore: Long document coreference with bounded memory neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8519–8526. find our card
  13. Christopher Williams and Matthias Seeger. 2001. Using the Nyström method to speed up kernel machines. In 2001. find
  14. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. Corefqa: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963. find our card
  15. Patrick Xia, João Sedoc, and Benjamin Van Durme. 2020. Incremental neural coreference resolution in constant memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8617–8624, Online. Association for Computational Linguistics. find our card
  16. Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533, Online. Association for Computational Linguistics. find our card

↑ back to top