Elamur Research
☰
home / papers / 2020 / Revealing the Myth of Higher-Order Inference in Coreference Resolution

Revealing the Myth of Higher-Order Inference in Coreference Resolution

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.686.pdf

Summary

Nearly every recent neural coreference resolver includes higher-order inference (HOI), a step that refines decisions using global, cluster-level information. Its actual value next to strong contextual encoders had rarely been tested. Xu and Choi reimplemented the end-to-end c2f-coref model of Lee et al. (2018) in PyTorch. They paired it with two existing HOI variants, attended antecedent and entity equalization, plus two of their own, span clustering and cluster merging. All systems were trained with BERT or SpanBERT encoders and evaluated on the English CoNLL 2012 shared task data, with five repeated runs each.

Moving from BERT to SpanBERT clearly helps. Adding HOI on top of SpanBERT, however, gives no benefit or even hurts. Only cluster merging edges ahead, reaching a best Avg-F1 of 80.2. Error analyses show that HOI fixes and breaks roughly the same number of links. Pronoun errors are also driven mainly by encoder quality. The authors conclude that encoder representations, not HOI, are behind recent progress, and that HOI needs to be rethought.

Key points

  • Four HOI methods were compared side by side on one shared system: AA, EE, SC and CM (SC and CM are new). The code is released at github.com/emorynlp/coref-hoi.
  • SpanBERT outperforms BERT by about 2.4 Avg-F1. Mean Avg-F1 over five runs:
    • SpanBERT without HOI: 79.7
    • +AA: 79.4
    • +EE: 78.9
    • +SC: 79.2
    • +CM: 79.9
  • The best single model (SpanBERT + CM) scores 80.2 Avg-F1 on the CoNLL 2012 English test set. The previous best c2f-coref-based result was 79.6 (Joshi et al., 2020).
  • Disabling HOI at test time reduces Avg-F1 by less than 0.2 for every method. About 1% of links change from wrong to correct, and about the same share change from correct to wrong.
  • Pronoun analysis:
    • SpanBERT cuts wrong pronoun–pronoun links by almost 10% compared with BERT.
    • EE and CM reduce false links by over 4%.
    • Clusters that mix singular and plural pronouns rarely involve ambiguous pronouns like "you", which undercuts the classic motivation for HOI.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

62 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 55 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.686 · arXiv 2009.12013

@inproceedings{xu2020revealing,
  title = {{Revealing the Myth of Higher-Order Inference in Coreference Resolution}},
  author = {Xu, Liyan and Choi, Jinho D.},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.686},
  eprint = {2009.12013},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.686}
}

References

16 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405–1415, Beijing, China. Association for Computational Linguistics. ↗ our card
  2. Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  4. Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. 2019. End-to-end deep reinforcement learning based coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 660–665, Florence, Italy. Association for Computational Linguistics. find
  5. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  6. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. ↗ our card
  7. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy. Association for Computational Linguistics. ↗
  8. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  9. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  10. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 104– 111, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
  11. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  12. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
  13. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1416–1426, Beijing, China. Association for Computational Linguistics. find our card
  14. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 994–1004, San Diego, California. Association for Computational Linguistics. find our card
  15. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics. ↗ our card
  16. Juntao Yu, Alexandra Uma, and Massimo Poesio. 2020. A cluster ranking model for full anaphora resolution. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 11–20, Marseille, France. European Language Resources Association. find