Elamur Research
☰
home / papers / 2020 / Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks

Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.685.pdf

Summary

Leading coreference systems need memory and runtime that grow quadratically with document length, which makes book-scale texts impractical. Entity-mention models that store only entity representations are cheaper, but they still keep every entity, and that set grows with the document. The authors observe that few entities are "active" (between their first and last mention) at any given token. Building on this, they design a quasi-online model that holds only a fixed number of entity slots, which keeps runtime linear in document length. It uses frozen SpanBERT-large encodings, a pretrained mention proposer, and sequential clustering of candidate spans. When memory is full, a learned module (LB-MEM) decides whether to evict a tracked entity, skip the mention, or mark it invalid. This is compared with an unbounded variant (U-MEM) and an LRU-based rule (RB-MEM) on LitBank and OntoNotes. With 20 slots, LB-MEM nearly matches U-MEM and clearly beats LRU eviction when memory is tight. It also lowers training memory. This shows that coreference models can scale to long documents without tracking every entity.

Key points

  • Entity statistics: the maximum number of entities active at one token is far below the total per document (LitBank 18 vs 199; OntoNotes 24 vs 94).
  • LitBank test CoNLL F1:
    • U-MEM reaches 75.9, a new state of the art (previous best, Bamman et al. 2020: 68.1).
    • LB-MEM with 5/10/20 cells scores 69.5/74.9/75.7.
    • RB-MEM with 5/10/20 cells scores 66.7/71.8/72.6.
  • OntoNotes test CoNLL F1:
    • U-MEM scores 77.4 and LB-MEM with 20 cells scores 77.3, both close to Xia et al. 2020 (78.2).
    • Both remain below Joshi et al. 2020 (79.6) and Wu et al. 2020 (83.1).
  • On a LitBank split, peak training memory drops from 11.6 GB (U-MEM) to about 8.0–9.1 GB, and inference is slightly faster. Peak inference memory is about the same (3.1 vs 3.2 GB) because the SpanBERT encoder dominates it.
  • LB-MEM ignores far fewer mentions than RB-MEM (e.g., 18.3 vs 83.2 on LitBank with 5 cells). This suggests it learns which entities will be mentioned again. Code is available at github.com/shtoshni92/long-doc-coref.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

43 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 39 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.685 · arXiv 2010.02807

@inproceedings{toshniwal2020learning,
  title = {{Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks}},
  author = {Toshniwal, Shubham and Wiseman, Sam and Ettinger, Allyson and Livescu, Karen and Gimpel, Kevin},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.685},
  eprint = {2010.02807},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.685}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Baddeley. 1986. Working Memory. Oxford University Press. find
  2. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An Annotated Dataset of Coreference in English Literature. In LREC. find our card
  3. Kevin Clark and Christopher D. Manning. 2015. Entity-Centric Coreference Resolution with Model Stacking. In ACL. find our card
  4. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. TACL, 8. find our card
  5. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for Coreference Resolution: Baselines and Analysis. In EMNLP. find our card
  6. Frank Keller. 2010. Cognitively Plausible Models of Human Language Processing. In ACL. find
  7. Jack W. Rae, Jonathan J. Hunt, Ivo Danihelka, Timothy Harley, Andrew W. Senior, Gregory Wayne, Alex Graves, and Tim Lillicrap. 2016. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes. In NeurIPS. find
  8. Altaf Rahman and Vincent Ng. 2011. Narrowing the modeling gap: a cluster-ranking approach to coreference resolution. Journal of Artificial Intelligence Research, 40:469–521. find
  9. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. 2016. One-shot Learning with Memory-Augmented Neural Networks. In ICML. find
  10. Veselin Stoyanov and Jason Eisner. 2012. Easy-first Coreference Resolution. In COLING. find
  11. MK Tanenhaus, MJ Spivey-Knowlton, KM Eberhard, and JC Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217). find
  12. Shubham Toshniwal, Allyson Ettinger, Kevin Gimpel, and Karen Livescu. 2020. PeTra: A Sparsely Supervised Memory Model for People Tracking. In ACL. find
  13. Kellie Webster and James R. Curran. 2014. Limited memory incremental coreference resolution. In COLING. find
  14. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning Global Features for Coreference Resolution. In NAACL. find our card
  15. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR. find
  16. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. Coreference Resolution as Query-based Span Prediction. In ACL. find
  17. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-Driven Analysis of Challenges in Coreference Resolution. In EMNLP. find our card
  18. Patrick Xia, João Sedoc, and Benjamin Van Durme. 2020. Revisiting Memory-Efficient Incremental Coreference Resolution. In EMNLP. find
  19. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end Neural Coreference Resolution. In EMNLP. find our card
  20. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-Order Coreference Resolution with Coarse-to-Fine Inference. In NAACL-HLT. find our card
  21. Fei Liu, Luke Zettlemoyer, and Jacob Eisenstein. 2019. The Referential Reader: A Recurrent Entity Network for Anaphora Resolution. In ACL. find
  22. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree. In ACL. find
  23. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, CoNLL ’12. *Figure 1: Histograms of Maximum Active Entities for documents in LitBank and OntoNotes.* *Figure 2: Histograms of Entity Spread as fraction of document length for LitBank and OntoNotes.* find our card