Elamur Research⌕
☰
home / papers / 2021 / End-to-end Neural Coreference Resolution Revisited: A Simple yet Effective Baseline

End-to-end Neural Coreference Resolution Revisited: A Simple yet Effective Baseline

Paper · 2021 · Tuan Manh Lai, Trung Bui and Doo Soon Kim

Summary

The paper asks whether the many extensions built on top of e2e-coref, the first end-to-end neural coreference model, are needed once a strong pretrained Transformer serves as the encoder. These extensions include higher-order inference, entity equalization and reinforcement learning on metrics. The authors strip the original architecture down. They use a SpanBERT encoder in place of GloVe, Turian and character embeddings, and they drop the span-width, genre and distance features. Speaker identity is no longer a binary feature: the speaker's name is written directly in front of each utterance. They add one training step. The mention scorer is first pretrained as a binary classifier on the annotated (non-singleton) mentions, and then the whole system is trained jointly on the usual antecedent likelihood. On the CoNLL-2012 English OntoNotes benchmark, this reduced model slightly beats every compared extended system, reaching 79.7 average F1. The result suggests that added architectural complexity should be justified against a strong simple baseline.

Key points

  • Data: the CoNLL-2012 Shared Task English data built on OntoNotes, split 2802/343/348 documents for train/dev/test. Evaluation reports MUC, B³ and CEAF φ4, plus their averaged F1.
  • Encoder: spanbert-large-cased, with separate learning rates of 5e-05 for the encoder and 1e-4 for the upper layers. Training ran 100 epochs at batch size 32 on two 16GB V100 GPUs, using gradient checkpointing and accumulation.
  • Results: 85.4 MUC F1, 78.7 B³ F1 and 75.0 CEAF φ4 F1, for 79.7 average. The strongest rival, c2f-coref + SpanBERT-large, scores 79.6 average and keeps a small edge only on CEAF φ4 (75.3).
  • Mention pruning: thanks to the pretrained extractor, the model keeps 0.25·n spans instead of 0.4·n. That is about 142 instead of about 200 candidates per document. Gold-mention recall on dev still rises, from 92.7% to 95.7%.
  • Long documents are cut into overlapping 512-token windows. Each token takes its representation from the window where it has the most surrounding context.

Interesting ideas

  • I1. A mention proposer can be warmed up on the coreference annotations alone, as a plain yes/no detector, before joint training; this yields both higher recall and a smaller candidate pool for the linker.
  • I2. Speaker identity is handled by writing the speaker's name into the text next to the utterance, so the encoder learns it like any other context instead of through a hand-made same-speaker feature.
  • I3. The pair score is the sum of mention-ness for each span plus a pairwise compatibility term, and the "no antecedent" option is pinned at zero; every link decision therefore splits into readable parts.
  • I4. Per-stage recall of the pruning step (gold mentions kept versus candidates proposed) is reported as its own evaluation, separate from end-task metrics.
  • I5. Long inputs are encoded in half-overlapping windows, and each token keeps the representation from the window where it sits most centrally.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Our coreference step could train or calibrate its mention-candidate filter separately on chain-member mentions. That would let the linker see fewer, better candidates, and the filter's recall could be checked on its own.
  • I2 (frames): In fable and tale dialogues, we could attach the speaker entity to each quoted utterance before resolving "I", "you" and "we". This turns speaker-based pronoun resolution into an explicit, traceable link.
  • I3 (frames): Coreference explanations could report the three components separately: is this a mention, is that a mention, and do they match. A rejected link would then show which factor caused it.
  • I4 (learn): Our evaluation could log, for each pipeline filter, how many gold items survive and how many candidates pass. This would show whether errors come from early pruning or from later decisions.
  • I5 (tree): gparse could use this trick on documents or sentences longer than the transformer limit. Every token would keep balanced left and right context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations🏢Adobe SystemsSan Jose · United States14 works · 3,493 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Coreference resolution · 550 works

Cite

Cite as Tuan Manh Lai, Trung Bui and Doo Soon Kim. 2021. End-to-end Neural Coreference Resolution Revisited: A Simple yet Effective Baseline. arXiv:2107.01700.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2107.01700 · arXiv 2107.01700

@misc{lai2021end,
  title = {{End-to-end Neural Coreference Resolution Revisited: A Simple yet Effective Baseline}},
  author = {Tuan Manh Lai, Trung Bui and Doo Soon Kim},
  year = {2021},
  doi = {10.48550/arxiv.2107.01700},
  eprint = {2107.01700},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2107.01700}
}

References

29 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vincent Ng, “Supervised noun phrase coreference research: The first fifteen years,” in ACL, 2010.
  2. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nate Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher D. Manning, “A multi-pass sieve for coreference resolution,” in EMNLP, 2010.
  3. Greg Durrett and Dan Klein, “Easy victories and uphill battles in coreference resolution,” in EMNLP, 2013.
  4. Kevin Clark and Christopher D. Manning, “Entity-centric coreference resolution with model stacking,” in ACL, 2015.
  5. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber, “Learning global features for coreference resolution,” ArXiv, vol. abs/1604.03035, 2016. ↗
  6. Kevin Clark and Christopher D. Manning, “Deep reinforcement learning for mention-ranking coreference models,” in EMNLP, 2016.
  7. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer, “End-to-end neural coreference resolution,” in EMNLP, 2017.
  8. Rui Zhang, Cı́cero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir R. Radev, “Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering,” in ACL, 2018. ↗
  9. Kenton Lee, Luheng He, and Luke Zettlemoyer, “Higher-order coreference resolution with coarse-to-fine inference,” in NAACL-HLT, 2018.
  10. Jia-Chen Gu, Zhen-Hua Ling, and Nitin Indurkhya, “A study on improving end-to-end neural coreference resolution,” in CCL, 2018.
  11. Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li, “End-to-end deep reinforcement learning based coreference resolution,” in ACL, 2019.
  12. Ben Kantor and Amir Globerson, “Coreference resolution with entity equalization,” in ACL, 2019.
  13. Mandar Joshi, Omer Levy, Daniel S. Weld, and Luke Zettlemoyer, “Bert for coreference resolution: Baselines and analysis,” in EMNLP/IJCNLP, 2019.
  14. Fang Kong and Jian Fu, “Incorporating structural information for better coreference resolution,” in IJCAI, 2019. find
  15. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy, “Spanbert: Improving pre-training by representing and predicting spans,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 64–77, 2020. ↗
  16. Liyan Xu and Jinho D. Choi, “Revealing the myth of higher-order inference in coreference resolution,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Nov. 2020, pp. 8527–8533, Association for Computational Linguistics.
  17. Yuval Kirstain, Ori Ram, and Omer Levy, “Coreference resolution without span representations,” ArXiv, vol. abs/2101.00434, 2021. ↗
  18. Tuan Lai, Heng Ji, and ChengXiang Zhai, “Bert might be overkill: A tiny but effective biomedical entity linker based on residual convolutional neural networks,” arXiv preprint arXiv:2109.02237, 2021. ↗
  19. Wenjiong Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li, “Corefqa: Coreference resolution as query-based span prediction,” in ACL, 2020.
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT, 2019.
  21. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio, “Word representations: A simple and general method for semi-supervised learning,” in ACL, 2010.
  22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning, “Glove: Global vectors for word representation,” in EMNLP, 2014.
  23. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, vol. abs/1409.0473, 2015. ↗
  24. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang, “Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes,” in EMNLP-CoNLL Shared Task, 2012.
  25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” ArXiv, vol. abs/1910.01108, 2019. ↗
  26. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou, “Mobilebert: a compact task-agnostic bert for resource-limited devices,” in ACL, 2020.
  27. Tuan Manh Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara, “A simple but effective bert model for dialog state tracking on resource-limited systems,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 8034–8038. ↗
  28. Tuan Lai, Heng Ji, Trung Bui, Quan Hung Tran, Franck Dernoncourt, and Walter Chang, “A context-dependent gated module for incorporating symbolic semantics into event coreference resolution,” arXiv preprint arXiv:2104.01697, 2021. ↗
  29. Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, et al., “Resin: A dockerized schema-guided cross-document cross-lingual cross-media information extraction and event tracking system,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, 2021, pp. 133–143. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lai-2021-end-end-neural-coreference-resolution/