Elamur Research⌕
☰
home / papers / 2016 / Revisiting the Evaluation for Cross Document Event Coreference

Revisiting the Evaluation for Cross Document Event Coreference

Summary

Cross document event coreference (CDEC) asks whether event mentions in different documents describe the same real-world event. The authors examine how CDEC has been evaluated on ECB+, the only public corpus with cross document event coreference annotations. They find that earlier evaluation protocols are inconsistent and too lenient. The B&H setting scores each topic separately, so wrong links across topics go unpunished. The YCF setting splits evaluation by sub-topic and also drops singleton mentions. Both depend on the corpus's topic layout, which a real system never sees at test time.

The paper proposes two complementary settings, SIMPLE-CDEC and PURE-CDEC. The second isolates decisions made across documents from those made inside a single document. Under these settings, a baseline that only links mentions within documents no longer looks deceptively strong, and Lemma-δ turns out to be a better baseline than plain Lemma matching. Keeping singletons in the evaluation also exposed many annotation gaps in ECB+. This matters because unreliable metrics and noisy gold data make it impossible to judge real progress on corpus-level event aggregation.

Key points

  • ECB+ statistics: 982 documents, 43 topics, 86 sub-topics and 14,874 event mentions, of which 7,605 are singletons. The authors use the train/dev/test splits of Yang et al. (2015).
  • Prior settings ignore errors across topics (B&H) or across sub-topics and singletons (YCF). The authors recommend always reporting both SIMPLE-CDEC and PURE-CDEC.
  • In PURE-CDEC, only systems that actually make cross document links (Lemma, Lemma-δ, SAC) get non-zero MUC and Blanc-PW scores. Lemma-WD loses about 10% average F, versus roughly 6% for Lemma-δ and SAC.
  • SAC and Lemma-δ perform best overall. Lemma-δ is proposed as the standard baseline in place of Lemma.
  • An anchored SVM (following Goldberg and Elhadad, 2007) with an anchor-weight threshold of 0.95, followed by manual review, uncovered over 300 missing singleton links in ECB+. The authors also note partially annotated documents and released the error list.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{upadhyay2016revisiting,
  title = {{Revisiting the Evaluation for Cross Document Event Coreference}},
  author = {Upadhyay, Shyam and Gupta, Nitish and Christodoulopoulos, Christos and Roth, Dan},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1183/}
}

References

39 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allan, Jaime G Carbonell, George Doddington, Jonathan Yamron, and Yiming Yang. 1998. Topic detection and tracking pilot study final report. find
  2. Jun Araki, Zhengzhong Liu, Eduard H Hovy, and Teruko Mitamura. 2014. Detecting subevent structure for event coreference resolution. In LREC. find our card
  3. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference. find our card
  4. Amit Bagga and Breck Baldwin. 1999. Cross-document event coreference: Annotations, experiments, and observations. In the Workshop on Coreference and its Applications. find our card
  5. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In ACL-COLING. find our card
  6. Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In ACL. find our card
  7. Cosmin Adrian Bejan and Sanda Harabagiu. 2014. Unsupervised event coreference resolution. Computational Linguistics. find our card
  8. E. Bengtson and D. Roth. 2008. Understanding the value of features for coreference resolution. In EMNLP. find our card
  9. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In EMNLP. find
  10. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In ACL. find
  11. Kai-Wei Chang, Shyam Upadhyay, Ming-Wei Chang, Vivek Srikumar, and Dan Roth. 2015. IllinoisSL: A JAVA library for structured prediction. In Arxiv Preprint, volume abs/1509.07179. — [arXiv:1509.07179](https://arxiv.org/abs/1509.07179) find
  12. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In LREC. find our card
  13. Agata Cybulska and Piek Vossen. 2015. Translating granularity of event slots into features for event coreference resolution. In Workshop on Events. find our card
  14. Naomi Daniel, Dragomir Radev, and Timothy Allison. 2003. Sub-event based multi-document summarization. In the HLT-NAACL 03 on Text summarization workshop-Volume 5. find
  15. Dipanjan Das and Noah Smith. 2011. Semi-supervised frame-semantic parsing for unknown predicates. In ACL. find our card
  16. Q. Do, Y. Chan, and D. Roth. 2011. Minimally supervised event causality identification. In EMNLP. find
  17. Yoav Goldberg and Michael Elhadad. 2007. SVM model tampering and anchored learning: A case study in hebrew NP chunking. In ACL. find our card
  18. Ben Hachey, Joel Nothman, and Will Radford. 2014. Cheap and easy entity evaluation. In ACL. find
  19. Kevin Humphreys, Robert Gaizauskas, and Saliha Azzam. 1997. Event coreference for information extraction. In a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts. find our card
  20. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In EMNLP-CoNLL. find our card
  21. Zhengzhong Liu, Jun Araki, Eduard H Hovy, and Teruko Mitamura. 2014. Supervised within-document event coreference using information propagation. In LREC. find our card
  22. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An extension of BLANC to system mentions. find
  23. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In EMNLP. find our card
  24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL: System Demonstrations. find our card
  25. James Mayfield, David Alexander, Bonnie J Dorr, Jason Eisner, Tamer Elsayed, Tim Finin, Clayton Fink, Marjorie Freedman, Nikesh Garera, Paul McNamee, et al. 2009. Cross-document coreference resolution: A key technology for learning by reading. In AAAI Spring Symposium: Learning by Reading and Learni… find
  26. Srini Narayanan and Sanda Harabagiu. 2004. Question answering based on semantic structures. In COLING. find
  27. V. Ng and C. Cardie. 2002. Identifying anaphoric and non-anaphoric noun phrases to improve coreference resolution. In COLING. find
  28. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In CoNLL. find our card
  29. Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event detection and co-reference with minimal supervision. In EMNLP, page 11. ACL. find our card
  30. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In CoNLL. find our card
  31. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In ACL. find our card
  32. V. Punyakanok, D. Roth, and W. Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2). find our card
  33. William M Rand. 1971. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association. find
  34. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the rand index for coreference evaluation. Natural Language Engineering. find
  35. Rui Sun, Yue Zhang, Meishan Zhang, and Donghong Ji. 2015. Event-driven headline generation. In ACL. find
  36. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In the 6th conference on Message understanding. find our card
  37. Travis Wolfe, Mark Dredze, and Benjamin Van Durme. 2015. Predicate argument alignment using a global coherence model. In NAACL. find
  38. Bishan Yang, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent bayesian model for event coreference resolution. TACL. find
  39. Tongtao Zhang, Hongzhi Li, Heng Ji, and Shih-Fu Chang. 2015. Cross-document event coreference resolution based on cross-media features. In EMNLP. find

↑ back to top

This page in Markdown →