Elamur Research⌕
☰
home / papers / 2021 / Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora

Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora

Summary

Cross-document event coreference resolution (CDCR) requires a system to find event mentions across a document collection and group those that denote the same real-world event. The authors observe that every existing CDCR system has been built, trained and tested on only one corpus. This makes it doubtful that such systems would work reliably in downstream multi-document applications.

To test this, they define a shared evaluation over three corpora: ECB+, the Gun Violence Corpus (GVC), and the Football Coreference Corpus, which they reannotate at token level as FCC-T. They build an interpretable mention-pair system that pools features from earlier work and compare it with the neural model of Barhom et al. (2019). The feature-based system scores lower overall but performs consistently on all three corpora. The neural model succeeds only on data resembling ECB+.

Model introspection shows that each corpus relies on different coreference cues, and models trained on one corpus transfer poorly to the others. The central takeaway is that CDCR systems must be evaluated on several corpora.

Key points

  • New resource: FCC-T is a token-level reannotation and extension of FCC. The data, annotation guidelines and system code are public at github.com/UKPLab/cdcr-beyond-corpus-tailored.
  • ECB+ structure: ECB+ contains 982 news articles across 43 topics, annotated topic by topic with no cross-topic links. The analysis shows that several systems overfit to this structure.
  • Coreference cues differ by corpus: In ECB+, coreference decisions depend mostly on the event action. FCC-T and GVC are more balanced and also require understanding of event participants and time.
  • Cross-corpus transfer: Models trained on a single corpus lose much of their performance on unseen corpora. The authors also require that systems handle about 100k mentions within a day on one GPU.
  • Training-pair sampling: Hyperparameters c and k control how many mention pairs are sampled, with k governing the precision/recall balance. The chosen settings are c=8, k=8 for ECB+ and GVC, and c=2, k=8 for FCC-T.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

20 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00407 · arXiv 2011.12249

@article{bugert2021generalizing,
  title = {{Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora}},
  author = {Bugert, Michael and Reimers, Nils and Gurevych, Iryna},
  journal = {Computational Linguistics},
  year = {2021},
  doi = {10.1162/coli\_a\_00407},
  eprint = {2011.12249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/coli_a_00407}
}

References

46 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiba, Takuya, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’19, pages 2623–2631, New York, NY. find our card
  2. Artstein, Ron and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555–596. https://doi.org/10.1162/coli.07-034-R2 Bagga, Amit and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In 36th Annual M… ↗
  3. Barhom, Shany, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179–4189, Florence. find our card
  4. Bejan, Cosmin and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412–1422, Uppsala. find our card
  5. Bejan, Cosmin Adrian and Sanda Harabagiu. 2014. Unsupervised event coreference resolution. Computational Linguistics, 40(2):311–347. https://doi.org/10.116/COLI a 00174 Bojanowski, Piotr, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transac… ↗ our card
  6. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254. find our card
  7. Chang, Angel X. and Christopher Manning. 2012. SUTime: A library for recognizing and normalizing time expressions. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3735–3740, Istanbul. find
  8. Chen, Tianqi and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16. pages 785–794, ACM, New York, NY, USA. find
  9. Choubey, Prafulla Kumar and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2114–2123, Copenhagen. find our card
  10. Choubey, Prafulla Kumar and Ruihong Huang. 2018. Improving event coreference resolution by modeling correlations between event coreference chains and document topic structures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4… find our card
  11. Choubey, Prafulla Kumar, Kaushik Raju, and Ruihong Huang. 2018. Identifying the most dominant event in a news article by mining event coreference relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find our card
  12. Cremisini, Andres and Mark Finlayson. 2020. New insights into cross-document event coreference: Systematic comparison and a simplified approach. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 1–10, Online. find our card
  13. Cybulska, Agata and Piek Vossen. 2014a. Guidelines for ECB+ annotation of events and their coreference. In Technical Report. Technical Report NWR-2014-1, VU University Amsterdam. find
  14. Cybulska, Agata and Piek Vossen. 2014b. Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4545–4552, Reykjavik. find our card
  15. Cybulska, Agata and Piek Vossen. 2015. “Bag of events” approach to event coreference resolution. Supervised classification of event templates. International Journal of Computational Linguistics and Applications, 6(2):11–27. find
  16. Fisch, Adam, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. find
  17. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13, Hong Kong. find our card
  18. Gardner, Matt, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne. find our card
  19. Guo, Mandy, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2021. MultiReQA: A cross-domain evaluation for retrieval question answering models. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 94–104, Kyiv. find
  20. Guyon, Isabelle, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene selection for cancer classification using support vector machines. Machine Learning, 46:389–422. https://doi.org/10.1023/A:1012487302797 Hermans, Alexander, Lucas Beyer, and Bastian Leibe. 2017. In defense of the tripl… — [doi:10.1023/a:1012487302797](https://doi.org/10.1023/a:1012487302797) ↗
  21. Hovy, Eduard, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In Workshop on Events: Definition, Detection, Coreference, and Representation, pages 21–28, Atlanta, GA. find
  22. Joshi, Mandar, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10.1162/tacl a 00300 Kenyon-Dean, Kian, Jackie… ↗ our card
  23. Klie, Jan Christoph, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, p… find
  24. Krippendorff, Klaus. 1995. On the reliability of unitizing continuous data. Sociological Methodology, 25:47–76. find
  25. Lee, Heeyoung, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 489–5… find our card
  26. Lerer, Adam, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019. PyTorch-BigGraph: A large-scale graph embedding system. In Proceedings of the 2nd SysML Conference, Palo Alto, CA. find
  27. Lu, Jing and Vincent Ng. 2018. Event coreference resolution: A survey of two decades of research. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 5479–5486, Stockholm. find
  28. Luo, Xiaoqiang. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver. find our card
  29. Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60, Baltimore, MD. find our card
  30. Meged, Yehudit, Avi Caciularu, Vered Shwartz, and Ido Dagan. 2020. Paraphrasing vs coreferring: Two sides of the same coin. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4897–4907, Online. Mendes, Pablo N., Max Jakob, Andrés Garcı́a-Silva, and Christian Bizer. 2011… find our card
  31. Mirza, Paramita, Fariz Darari, and Rahmad Mahendra. 2018. KOI at SemEval-2018 task 5: Building knowledge graph of incidents. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 81–87, New Orleans, LA. find
  32. Moosavi, Nafise Sadat and Michael Strube. 2016. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin. find our card
  33. Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:28… find
  34. Postma, Marten, Filip Ilievski, and Piek Vossen. 2018. SemEval-2018 task 5: Counting events and participants in the long tail. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 70–80, New Orleans, LA. find
  35. Pradhan, Sameer, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pa… find our card
  36. Reimers, Nils and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  37. Rousseeuw, Peter J. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65. https://doi.org/10.1016 /0377-0427(87)90125-7 Shannaq, Boumedyen and Victor V. Alexandrov. 2010. Using product similarity for a… find
  38. Shi, Peng and Jimmy Lin. 2019. Simple BERT models for relation extraction and semantic role labeling. arXiv preprint, arXiv:1904.05255. Surdeanu, Mihai, Lluı́s Màrquez, Xavier Carreras, and Pere R Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelligence Res… find our card
  39. Upadhyay, Shyam, Nitish Gupta, Christos Christodoulopoulos, and Dan Roth. 2016. Revisiting the evaluation for cross document event coreference. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1949–1958, Osaka. find our card
  40. Van Landeghem, Sofie, Jari Bjrne, Chih-Hsuan Wei, Kai Hakala, Sampo Pyysalo, Sophia Ananiadou, Hung-Yu Kao, Zhiyong Lu, Tapio Salakoski, Yves Van de Peer, and Filip Ginter. 2013. Large-scale event extraction from literature with multi-level gene normalization. PLOS ONE, 8(4):1–12. https://doi.org/10… find
  41. Vossen, Piek. 2018. NewsReader at SemEval-2018 task 5: Counting events by reasoning over event-centric-knowledge-graphs. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 660–666, New Orleans, LA. find
  42. Vossen, Piek and Agata Cybulska. 2016. Identity and Granularity of Events in Text. In Computational Linguistics and Intelligent Text Processing, Cham: Springer International Publishing, pages 501–522. find
  43. Vossen, Piek, Filip Ilievski, Marten Postma, and Roxane Segers. 2018. Don’t annotate, but validate: A data-to-text method for capturing event data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 3034–3042, Paris. find
  44. Walker, Christopher, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  45. ACE 2005 multilingual training corpus. https://catalog.ldc.upenn.edu/LDC2006T06 Wright-Bettner, Kristin, Martha Palmer, Guergana Savova, Piet de Groen, and Timothy Miller. 2019. Cross-document coreference: An approach to capturing coreference without context. In Proceedings of the Tenth Internation… ↗
  46. Yang, Bishan, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent Bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3:517–528. https://doi.org/10.1162/tacl a 00155 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bugert-2021-generalizing-cross-document-event-coreference/