Elamur Research⌕
☰
home / papers / 2014 / Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution

Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution

Summary

This paper asks whether the EventCorefBank (ECB), a widely used benchmark for within- and cross-document event coreference resolution, reflects the variety found in real news streams. To test this, the authors built a baseline that links event action mentions only when their lemmas match, using NLTK tools and the WordNet lemmatizer. Despite having no entity, syntactic or anaphora information, the baseline scored close to published machine-learning systems. When matching was limited to mentions inside the same topic, precision went up sharply. The authors conclude that most ECB topics cover only one seminal event, so the task largely collapses into topic classification. They also find little lexical variation within coreference chains. To fix this, they collected 502 new news texts describing a second instance of each event type already present in the 43 ECB topics. They then annotated the event components (action, time, location, human and non-human participants) and their coreference. The resulting ECB+ corpus is harder and more realistic, because systems must tell apart different events of the same type. It is freely available.

Key points

  • The original ECB has 43 topics and 482 GoogleNews texts. ECB+ adds 502 texts, bringing the total to 982.
  • The cross-topic lemma-match baseline reached 54.04 MUC F, 59.54 B3 F, 40.00 CEAFm, 56.91 BLANC F and 52.52 CoNLL F1. Restricting matching to within-topic pairs raised CoNLL F1 to 64.76 and B3 precision to 92.75.
  • Annotation used 30 fine-grained tags grouped into 5 main event-component classes. Annotators did cross-document linking in the CROMER tool.
  • Inter-annotator agreement (Cohen's Kappa, topics 1–4, 55 texts) was 0.74 for all 30 tags and 0.79 for the 5 main classes. Counting only tokens tagged by at least one annotator gave 0.63 and 0.68. Cross-document coreference agreement was 0.76.
  • ECB+ can be downloaded from the NewsReader project site (newsreader-project.eu).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

108 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cybulska2014using,
  title = {{Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution}},
  author = {Cybulska, Agata and Vossen, Piek},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/56eog75zit7h},
  url = {https://doi.org/10.63317/56eog75zit7h}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bagga, Amit and Baldwin, Breck. (1998). Algorithms for scoring coreference chains. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). find our card
  2. Bartalesi Lenzi, Valentina, Moretti, Giovanni, and Sprugnoli, Rachele. (2012). CAT: the CELCT Annotation Tool. In Proceedings of LREC 2012. find
  3. Bejan, Cosmin Adrian and Harabagiu, Sanda. (2008). A linguistic resource for discovering event structures and resolving event coreference. In In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. find our card
  4. Bejan, Cosmin Adrian and Harabagiu, Sanda. (2010). Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden. find our card
  5. Bejan, Cosmin Adrian, Titsworth, Matthew, Hickl, Andrew, and Harabagiu, Sanda. (2009). Nonparametric bayesian models for unsupervised event coreference resolution. In Advances in Neural Information Processing Systems 22, pages 73–81. find
  6. Chen, Bin, Su, Jian, Pan, Sinno Jialin, and Tan, Chew Lim. (2011). A unified event coreference resolution by integrating multiple resolvers. In Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, November. find our card
  7. Cohen, J. (1960). The coefficient of agreement for nominal scales. Educational and Psychological Measurement, 21(1):37–46. find
  8. Cybulska, Agata and Vossen, Piek. (2014). Guidelines for ecb+ annotation of events and their coreference. Technical Report NWR-2014-1, VU University Amsterdam. find
  9. Landis, J. R. and Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  10. LDC. (2005). Ace (automatic content extraction) english annotation guidelines for events ver. 5.4.3 2005.07.01. Technical report, Linguistic Data Consortium. find
  11. Lee, Heeyoung, Recasens, Marta, Chang, Angel, Surdeanu, Mihai, and Jurafsky, Dan. (2012). Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing and Natural Language Learning (EMNLP-CoNLL). find our card
  12. Linguistic Data Consortium. (2008). Ace (automatic content extraction) english annotation guidelines for entities, version 6.6 2008.06.13. Technical report, June. http://projects.ldc.upenn.edu/ace/docs/English-Entities-Guidelines v6.6.pdf. ↗
  13. Luo, Xiaoqiang. (2005). On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (EMNLP-2005). find our card
  14. Pradhan, Sameer, Ramshaw, Lance, Weischedel, Ralph, MacBride, Jessica, and Micciulla, Linnea. (2007). Unrestricted coreference: Indentifying entities and events in ontonotes. In Proceedings of the IEEE International Conference on Semantic Computing (ICSC), September. find
  15. Pradhan, Sameer, Ramshaw, Lance, Marcus, Mitchell, Palmer, Martha, Weischedel, Ralph, and Xue, Nianwen. (2011). Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of CoNLL 2011: Shared Task. find our card
  16. Pustejovsky, James, Castano, Jose, Ingria, Bob, Sauri, Roser, Gaizauskas, Rob, Setzer, Andrea, and Katz, Graham. (2003). Timeml: Robust specification of event and temporal expressions in text. In Proceedings of Computational Semantics Workshop (IWCS-5). find
  17. Recasens, Marta and Hovy, Eduard. (2011). Blanc: Implementing the rand index for coreference evaluation. Natural Language Engineering, 17(4):485–510. find
  18. Recasens, Marta. (2011). Annotation guidelines for entity and event coreference. In http://www.bbn.com/NLP/OntoNotes. ↗
  19. Saurı́, Roser, Littman, Jessica, Knippen, Robert, Gaizauskas, Robert, Setzer, Andrea, and Pustejovsky, James. (2005). Timeml 1.2.1 annotation guidelines, October. http://timeml.org/site/publications/timeMLdocs/annguide 1.2.1.pdf. ↗
  20. Sinclair, John. (2004). Developing linguistic corpora: a guide to good practice. http://ota.ahds.ac.uk/documents/creating/dlc/chap-ter1.htm\#section4. ↗
  21. Vilain, Marc, Burger, John, Aberdeen, John, Connolly, Dennis, and Hirschman, Lynette. (1995). A model theoretic coreference scoring scheme. In Proceedings of MUC-6. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cybulska-2014-sledgehammer-crack-nut-lexical-diversity/