Elamur Research⌕
☰
home / papers / 2021 / Conundrums in Event Coreference Resolution: Making Sense of the State of the Art

Conundrums in Event Coreference Resolution: Making Sense of the State of the Art

Summary

Span-based neural models have recently set new records in event coreference resolution (CR), but it is unclear where their gains come from. Lu and Ng examine their own state-of-the-art span-based resolver (Lu and Ng, 2021), which jointly learns six tasks: trigger detection, argument extraction, attribute prediction, anaphoricity prediction, entity CR and event CR. Soft cross-task consistency constraints link these tasks. The authors test the system on ACE 2005 and KBP 2017. They compare encoders (SpanBERT-base vs. SpanBERT-large), six-task "Knowledge-Rich" vs. two-task "Knowledge-Lean" setups, and pipeline vs. joint training. They also run perturbation tests and oracle experiments. The strongest configuration combines joint training, rich knowledge and SpanBERT-large. Two kinds of anaphoric mentions remain hard: those that lemma matching cannot resolve and those whose mention pairs never appear in training. The resolver reacts to every perturbation tested, but no single one stands out. The oracle studies show that better mention boundaries, anaphoricity and especially event subtyping would bring the largest gains. The analysis gives future work a concrete map of priorities.

Key points

  • Datasets: ACE 2005 has 33 subtypes, 30 roles and 4 attributes, and uses the Wadden et al. (2019) split. KBP 2017 has 18 subtypes, 20 roles and 1 attribute; its training data comes from LDC2015E29/E68/E73/E94 and LDC2016E64. ECB+ was left out because its within-document annotation is incomplete.
  • Evaluation: The official KBP 2017 scorer (v1.8) reports AVG-F, the mean F-score over MUC, B3, CEAFe and BLANC.
  • Best results: The joint Knowledge-Rich model with SpanBERT-l reaches 60.1 AVG-F on ACE and 48.0 on KBP.
  • Single oracles: Perfect anaphoricity adds 4.7–5.8 AVG-F points. Gold subtypes add 17.9–21.2 points, partly because the scorer requires correct subtypes.
  • Combined and agreement oracles: Gold boundaries plus gold subtypes push AVG-F above 80% on both datasets. Agreement oracles (subtype, argument and attribute agreement) help only marginally.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.103

@inproceedings{lu2021conundrums,
  title = {{Conundrums in Event Coreference Resolution: Making Sense of the State of the Art}},
  author = {Lu, Jing and Ng, Vincent},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.103},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.103}
}

References

41 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events, pages 1–8, Sydney, Australia. Association for Computational Linguistics. find
  2. Jun Araki, Zhengzhong Liu, Eduard Hovy, and Teruko Mitamura. 2014. Detecting subevent structure for event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. European Language Resources Association (ELRA). find our card
  3. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, pages 563–566. find our card
  4. Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412–1422, Uppsala, Sweden. Association for Computational Linguistics. find our card
  5. Cosmin Adrian Bejan and Sanda Harabagiu. 2014. Unsupervised event coreference resolution. Computational Linguistics, 40(2):311–347. find our card
  6. Chen Chen and Vincent Ng. 2013. Chinese event coreference resolution: Understanding the state of the art. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 822–828, Nagoya, Japan. Asian Federation of Natural Language Processing. find our card
  7. Chen Chen and Vincent Ng. 2014. SinoCorefer-encer: An end-to-end Chinese event coreference resolver. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4532–4538, Reykjavik, Iceland. European Language Resources Association (ELRA). find
  8. Chen Chen and Vincent Ng. 2015. Chinese event coreference resolution: An unsupervised probabilistic model rivaling supervised resolvers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1097–1107,… find our card
  9. Chen Chen and Vincent Ng. 2016. Joint inference over a lightly supervised information extraction pipeline: Towards event coreference resolution for resource-scarce languages. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 2913– 2920. AAAI Press. find
  10. Zheng Chen and Heng Ji. 2009. Graph-based event coreference resolution. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4), pages 54–57, Suntec, Singapore. Association for Computational Linguistics. find our card
  11. Zheng Chen, Heng Ji, and Robert Haralick. 2009. A pairwise event coreference model, feature impact and evaluation for event coreference resolution. In Proceedings of the Workshop on Events in Emerging Text Types, pages 17–22, Borovets, Bulgaria. Association for Computational Linguistics. find our card
  12. Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124–2133, Copenhagen, Denmark. Association for Computational Lingui… find our card
  13. Prafulla Kumar Choubey and Ruihong Huang. 2018. Improving event coreference resolution by modeling correlations between event coreference chains and document topic structures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 48… find our card
  14. Prafulla Kumar Choubey and Ruihong Huang. 2021. Automatic data acquisition for event coreference resolution. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1185– 1196, Online. Association for Computational Linguistic… find
  15. Agata Cybulska and Piek Vossen. 2015a. "Bag of events" approach to event coreference resolution. Supervised classification of event templates. International Journal of Computational Linguistics and Applications, 6(2):11–27. find
  16. Agata Cybulska and Piek Vossen. 2015b. Translating granularity of event slots into features for event coreference resolution. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1–10, Denver, Colorado. Association for Computational Linguist… find our card
  17. Yin Jou Huang, Jing Lu, Sadao Kurohashi, and Vincent Ng. 2019. Improving event coreference resolution by learning argument compatibility from unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  18. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… find our card
  19. Sebastian Krause, Feiyu Xu, Hans Uszkoreit, and Dirk Weissenborn. 2016. Event linking with sentential features from convolutional neural networks. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 239–249, Berlin, Germany. Association for Computational Li… find
  20. LDC. 2005. ACE (Automatic Content Extraction) English annotation guidelines for events Version 5.4.3. Linguistic Data Consortium. find
  21. LDC. 2016. Rich ERE annotation guidelines overview V4.2. Linguistic Data Consortium. find
  22. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  23. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–… find our card
  24. Zhengzhong Liu, Jun Araki, Eduard Hovy, and Teruko Mitamura. 2014. Supervised within-document event coreference using information propagation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4539–4544, Reykjavik, Iceland. European Language R… find our card
  25. Jing Lu and Vincent Ng. 2016. Event coreference resolution with multi-pass sieves. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3996– 4003, Portorož, Slovenia. European Language Resources Association (ELRA). find
  26. Jing Lu and Vincent Ng. 2017. Joint learning for event coreference resolution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 90–101, Vancouver, Canada. Association for Computational Linguistics. find
  27. Jing Lu and Vincent Ng. 2020. Event coreference resolution with non-local information. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 653–663, Suzhou,… find
  28. Jing Lu and Vincent Ng. 2021. Constrained multi-task learning for event coreference resolution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4504–4514, Online. Association for Computational Li… find our card
  29. Jing Lu, Deepak Venugopal, Vibhav Gogate, and Vincent Ng. 2016. Joint inference for event coreference resolution. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3264–3275, Osaka, Japan. find our card
  30. The COLING 2016 Organizing Committee. find
  31. Yaojie Lu, Hongyu Lin, Jialong Tang, Xianpei Han, and Le Sun. 2020. End-to-end neural event coreference resolution. CoRR, abs/2009.08153. — [arXiv:2009.08153](https://arxiv.org/abs/2009.08153) find
  32. Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… find our card
  33. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. find our card
  34. Katie McConky, Rakesh Nagi, Moises Sudit, and William Hughes. 2012. Improving event coreference by context extraction and dynamic feature weighting. In CogSIMA, pages 38–43. find
  35. Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event detection and co-reference with minimal supervision. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 392–402, Austin, Texas. Association for Computational Linguistics. find our card
  36. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  37. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. find
  38. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language… find our card
  39. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. find our card
  40. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… find our card
  41. Bishan Yang, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent Bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3:517–528. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2021-conundrums-event-coreference-resolution-making/