Elamur Research⌕
☰
home / papers / 2020 / Streamlining Cross-Document Coreference Resolution: Evaluation and Modeling

Streamlining Cross-Document Coreference Resolution: Evaluation and Modeling

Paper · 2020 · Arie Cattan, Alon Eirew, Gabriel Stanovsky et al.

Summary

Cross-document (CD) coreference resolution links mentions of the same entity or event across separate texts. Its progress has lagged behind within-document (WD) coreference, partly because evaluations on ECB+ have been lenient and inconsistent. Many works fed gold mentions to the model, scored singleton clusters, or worked on gold or near-perfect subtopics, which removes the ambiguity ECB+ was built to test. The authors propose a stricter protocol modelled on the OntoNotes WD standard. Systems start from raw text, singletons are excluded from scoring, and results are reported at both topic and full-corpus level. To give this protocol a baseline, they build what they describe as the first end-to-end CD coreference model. It adapts recent neural WD architectures with a RoBERTa encoder, groups mentions through agglomerative clustering, and uses no external resources. Under the older protocols it beats prior state of the art by about 3 CoNLL F1. Under the realistic setup its scores fall sharply, which shows that earlier figures were inflated and that mention detection and linking both have much room to improve.

Key points

  • Benchmark and code: Experiments use ECB+, whose topics each contain two event-based subtopics with similar vocabulary. The code is publicly released (github.com/ariecattan/coref).
  • Predicted mentions, no singletons: CoNLL F1 at topic/corpus level is 48.2/46.9 for events, 34.4/34.2 for entities and 41.4/40.7 for the joint "ALL" setting. With gold mentions the corresponding topic-level scores are 62.0, 65.3 and 63.4.
  • Subtopic vs. topic evaluation: Moving from subtopic to topic level costs much more for events (71.1 → 62.0) than for entities (67.3 → 65.3). A preliminary document clustering step grouped the 10 gold test topics into 8 predicted ones with little loss.
  • Ablation (dev set, event, topic level, base 58.1 F1): Removing RoBERTa costs 4.1 points, dynamic pruning 4.0, mention-scorer pre-training 3.2 and negative sampling 1.4.
  • Error analysis: The main errors come from over-reliance on lexical overlap, failure to merge differently worded mentions, and weak temporal reasoning (e.g., not linking "Today" in one article to "Saturday" in another).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States422 works · 213,538 citations🎓Bar-Ilan UniversityRamat Gan · Israel150 works · 25,428 citations🎓Hebrew University of JerusalemJerusalem · Israel71 works · 9,675 citations🏢IntelSanta Clara · United States22 works · 43,146 citations🏢Intel (Israel)Kiryat Gat · Israel8 works · 495 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Coreference resolution · 576 works

Cite

Cite as Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi and Ido Dagan. 2020. Streamlining Cross-Document Coreference Resolution: Evaluation and Modeling. arXiv:2009.11032.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2009.11032 · arXiv 2009.11032

@misc{cattan2020streamlining,
  title = {{Streamlining Cross-Document Coreference Resolution: Evaluation and Modeling}},
  author = {Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi and Ido Dagan},
  year = {2020},
  doi = {10.48550/arxiv.2009.11032},
  eprint = {2009.11032},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2009.11032}
}

References

34 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document core f erencing using the vector space model. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 79–85, Montreal, Quebec, Canada. Assoc… find
  2. Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179–4189, Florence, Ital…
  3. Cosmin Bejan and Sanda Harabagiu. 2008. A linguistic resource for discovering event structures and resolving event coreference. In LREC 2008.
  4. Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412–1422, Uppsala, Sweden. Association for Computational Linguistics.
  5. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172–181, Brussels, Belgium. Association for Computational…
  6. Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124–2133, Copenhagen, Denmark. Association for Computational Lingui…
  7. Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163–1173, Atlanta, Georgia.… find
  8. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pages 4545– 4552, Reykjavik, Iceland. European Languages Resources Ass…
  9. Agata Cybulska and Piek Vossen. 2015. Translating granularity of event slots into features for event coreference resolution. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1–10, Denver, Colorado. Association for Computational Linguisti…
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2018. Neural models for reasoning over multiple mentions using coreference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie…
  12. Tobias Falke, Christian M. Meyer, and Iryna Gurevych. 2017. Concept-map-based multi-document summarization using concept coreference resolution and global importance optimization. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), page…
  13. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. find
  14. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77.
  15. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP…
  16. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy. Association for Computational Linguistics.
  17. Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 1–10, New Orleans, Louisiana. Asso…
  18. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  19. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 489–50…
  20. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  21. Kenton Lee, Luheng He, and Luke S. Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  23. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
  24. Y. Meged, Avi Caciularu, Vered Shwartz, and I. Dagan. 2020. Paraphrasing vs coreferring: Two sides of the same coin. ArXiv, abs/2004.14979.
  25. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Associ…
  26. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. find
  27. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti…
  28. Vered Shwartz, Gabriel Stanovsky, and Ido Dagan. 2017. Acquiring predicate paraphrases from news tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 155–160, Vancouver, Canada. Association for Computational Linguistics. find
  29. Shyam Upadhyay, Nitish Gupta, Christos Christodoulopoulos, and Dan Roth. 2016. Revisiting the evaluation for cross document event coreference. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1949–1958, Osaka, Japan.
  30. The COLING 2016 Organizing Committee. find
  31. Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. 2019. Do multi-hop readers dream of reasoning chains? In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 91–97, Hong Kong, China. Association for Computational Linguistics. find
  32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  33. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics.
  34. Bishan Yang, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent Bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3:517–528. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cattan-2020-streamlining-cross-document-coreference-resolution/