Elamur Research
☰
home / papers / 2020 / MAVEN: A Massive General Domain Event Detection Dataset

MAVEN: A Massive General Domain Event Detection Dataset

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.129.pdf

Summary

Event detection (ED) means finding the trigger words that signal events and assigning each an event type. Its standard benchmarks are small and narrow. ACE 2005 has only 599 documents and 33 subtypes, and most of those subtypes have very few examples. That is too little to train or reliably compare modern neural models, and it leaves most general-domain events uncovered. The authors address this with MAVEN, a large human-annotated ED dataset built from the introductory sections of English Wikipedia articles indexed by EventWiki. Its event types come from FrameNet frames, which were simplified and organized into a tree-shaped hierarchy so that crowd workers could annotate them. The authors re-implemented recent state-of-the-art ED models and evaluated them on MAVEN. These models perform noticeably worse there than on older benchmarks, which shows that general-domain ED remains unsolved. Further analyses point to useful directions: modeling several events in one sentence, using the type hierarchy to separate similar types, and transfer learning for low-resource ED settings.

Key points

  • Scale: MAVEN contains 4,480 documents, about 1,276k tokens, 49,873 sentences, 111,611 events and 118,732 event mentions. That is roughly twenty times the size of ACE 2005.
  • Event types: The schema has 168 types. They were derived from 598 event-related FrameNet frames, covering 74.4% of them, and arranged hierarchically using FrameNet inheritance and HowNet.
  • Topic coverage: The documents span 90 of the 95 major EventWiki topics. Military conflict is the largest topic, accounting for 32.5% of documents.
  • Error analysis: Trigger identification errors (false positives and false negatives) are the most frequent mistakes. Type confusions occur mostly between parent/child or sibling types and toward frequent classes.
  • Transfer to ACE 2005: Adding MAVEN directly as extra training data hurts performance on ACE 2005. Using MAVEN for intermediate pre-training improves it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Semantic role labeling

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.129

@inproceedings{wang2020maven,
  title = {{MAVEN: A Massive General Domain Event Detection Dataset}},
  author = {Wang, Xiaozhi and Wang, Ziqi and Han, Xu and Jiang, Wangyi and Han, Rong and Liu, Zhiyuan and Li, Juanzi and Li, Peng and Lin, Yankai and Zhou, Jie},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.129},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.129}
}

References

70 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacqueline Aguilar, Charley Beller, Paul McNamee, Benjamin Van Durme, Stephanie Strassel, Zhiyi Song, and Joe Ellis. 2014. A comparison of the events and relations across ACE, ERE, TAC-KBP, and FrameNet annotation standards. In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Cor… find
  2. David Ahn. 2006. The stages of event extraction. In Proceedings of ACL Workshop on Annotating and Reasoning about Time and Events, pages 1–8. find
  3. James Allan. 2012. Topic detection and tracking: event-based information organization, volume 12. Springer Science & Business Media. find
  4. Jun Araki and Teruko Mitamura. 2015. Joint event trigger identification and event coreference resolution with structured perceptron. In Proceedings of EMNLP, pages 2074–2080. find
  5. Jun Araki and Teruko Mitamura. 2018. Open-domain event detection using distant supervision. In Proceedings of COLING, pages 878–891. find
  6. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of ACL-COLING, pages 86–90. find our card
  7. P Basile, A Caputo, G Semeraro, and L Siciliani. 2014. Extending an information retrieval system through time event extraction. In Proceedings of DART, pages 36–47. find
  8. Steven Bird. 2006. NLTK: The Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72. find
  9. Ekaterina Buyko, Elena Beisswanger, and Udo Hahn. 2010. The GeneReg corpus for gene expression regulation events — an overview of the corpus and its in-domain and out-of-domain interoperability. In Proceedings of LREC. find
  10. Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu, and Jun Zhao. 2017. Automatically labeled data generation for large scale event extraction. In Proceedings of ACL, pages 409–419. find
  11. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of ACL-IJCNLP, pages 167–176. find
  12. Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yantao Jia. 2018. Collective event detection via a hierarchical and bias tagging networks with gated multi-level attention mechanisms. In Proceedings of EMNLP, pages 1267–1276. find
  13. Pengxiang Cheng and Katrin Erk. 2018. Implicit argument prediction with event knowledge. In Proceedings of ACL, pages 831–840. find
  14. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  15. Linguistic Data Consortium. 2005. ACE (Automatic Content Extraction) English annotation guidelines for events. Version, 5(4). find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186. find
  17. Ning Ding, Ziran Li, Zhiyuan Liu, Haitao Zheng, and Zibo Lin. 2019. Event detection with trigger-aware lattice neural network. In Proceedings of EMNLP-IJCNLP, pages 347–356. find
  18. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program – tasks, data, and evaluation. In Proceedings of LREC. find
  19. Zhendong Dong and Qiang Dong. 2003. HowNet - a hybrid language and knowledge resource. In Proceedings of NLP-KE, pages 820–824. find
  20. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2015. Overview of linguistic resources for the TAC KBP 2015 evaluations: Methodologies and results. In TAC. find
  21. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2016. Overview of linguistic resources for the TAC KBP 2016 evaluations: Methodologies and results. In TAC. find
  22. Joe Ellis, Jeremy Getman, and Stephanie M Strassel. 2014. Overview of linguistic resources for the TAC KBP 2014 evaluations: Planning, execution, and results. In TAC. find
  23. Xiaocheng Feng, Lifu Huang, Duyu Tang, Bing Qin, Heng Ji, and Ting Liu. 2016. A language-independent neural network for event detection. In Proceedings of ACL, pages 66–71. find
  24. James Ferguson, Colin Lockard, Daniel Weld, and Hannaneh Hajishirzi. 2018. Semi-supervised event extraction with paraphrase clusters. In Proceedings of NAACL, pages 359–364. find
  25. Charles Fillmore. 2006. Frame semantics. Cognitive linguistics: Basic readings, 34:373–400. find
  26. Charles J Fillmore. 1976. Frame semantics and the nature of language. In Annals of the New York Academy of Sciences: Conference on the origin and development of language and speech, volume 280, pages 20–32. find
  27. Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, and Ming Zhou. 2018. EventWiki: A knowledge base of major events. In Proceedings of LREC. find
  28. Jeremy Getman, Joe Ellis, Zhiyi Song, Jennifer Tracey, and Stephanie M Strassel. 2017. Overview of linguistic resources for the TAC KBP 2017 evaluations: Methodologies and results. In TAC. find
  29. Reza Ghaeini, Xiaoli Fern, Liang Huang, and Prasad Tadepalli. 2016. Event nugget detection with forward-backward recurrent neural networks. In Proceedings of ACL, pages 369–373. find
  30. Ralph Grishman and Beth Sundheim. 1996. Message understanding conference- 6: A brief history. In Proceedings of COLING. find
  31. Weiwei Guo, Hao Li, Heng Ji, and Mona Diab. 2013. Linking tweets to news: A framework to enrich short text data in social media. In Proceedings of ACL, pages 239–249. find
  32. Prashant Gupta and Heng Ji. 2009. Predicting unknown time arguments based on cross-event propagation. In Proceedings of ACL-IJCNLP, pages 369– 372. find
  33. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  34. Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, Clare R. Voss, Jiawei Han, and Avirup Sil. 2016. Liberal event extraction and event schema induction. In Proceedings of ACL, pages 258–268. find
  35. Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018. Zero-shot transfer learning for event extraction. In Proceedings of ACL, pages 2160–2170. find
  36. Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL, pages 254–262. find
  37. Jin-Dong Kim, Tomoko Ohta, Kanae Oda, and Jun’ichi Tsujii. 2008. From Text to Pathway: Corpus annotation for knowledge acquisition from biomedical literature. Series on Advances in Bioinformatics and Computational Biology, 6:165–176. find
  38. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  39. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML, page 282289. find
  40. Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of ACL, pages 73–82. find
  41. Wei Li, Dezhi Cheng, Lei He, Yuanzhuo Wang, and Xiaolong Jin. 2019. Joint event extraction based on hierarchical event schemas from framenet. IEEE Access, 7:25001–25015. find
  42. Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019. Cost-sensitive regularization for label confusion-aware event detection. In Proceedings of ACL, pages 5278–5283. find
  43. Shulin Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2017. Exploiting argument information to improve event detection via supervised attention mechanisms. In Proceedings of ACL, pages 1789–1798. find
  44. Claire Nédellec, Robert Bossy, Jin-Dong Kim, Jung-jae Kim, Tomoko Ohta, Sampo Pyysalo, and Pierre Zweigenbaum. 2013. Overview of BioNLP shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 1–7. find
  45. Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of NAACL, pages 300–309. find
  46. Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of ACL, pages 365– 371. find
  47. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543. find
  48. James Pustejovsky, Patrick Hanks, Roser Saur, Andrew See, Rob Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. 2003. The TimeBank corpus. Proceedings of Corpus Linguistics. find
  49. Sampo Pyysalo, Filip Ginter, Juho Heimonen, Jari Bjrne, Jorma Boberg, Jouni Jrvinen, and Tapio Salakoski. 2007. BioInfer: A corpus for information extraction in the biomedical domain. BMC bioinformatics, 8:50. find
  50. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383–2392. find our card
  51. Xiao Liu, Heyan Huang, and Yue Zhang. 2019. Open domain event extraction using neural latent variable models. In Proceedings of ACL, pages 2860–2871. find
  52. Alan Ritter, Mausam Mausam, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. Proceedings of the ACM SIGKDD, 1104-1112. find
  53. Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018. Jointly multiple events extraction via attention-based graph information aggregation. In Proceedings of EMNLP, pages 1247–1256. find
  54. Matthew Sims, Jong Ho Park, and David Bamman. 2019. Literary event detection. In Proceedings of ACL, pages 3623–3634. find
  55. Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. 2019. Distilling discrimination and generalization knowledge for event detection via delta-representation learning. In Proceedings of ACL, pages 4366–4376. find
  56. Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: Annotation of entities, relations, and events. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference,… find
  57. Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begoña Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the NewsReader multilingual event and time corpus. In Proceedings of LREC, pages 4417–4422. find
  58. Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A Smith. 2017. Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold. arXiv preprint arXiv:1706.09528. find our card
  59. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL, pages 1003–1011. find
  60. Paul Thompson, Syed Iqbal, John McNaught, and Sophia Ananiadou. 2009. Construction of an annotated corpus to support biomedical information extraction. BMC bioinformatics, 10:349. find
  61. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  62. ACE 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57. find
  63. Ying Zeng, Yansong Feng, Rong Ma, Zheng Wang, Rui Yan, Chongde Shi, and Dongyan Zhao. 2018. Scale up event extraction learning via automatic training data generation. In Proceedings of AAAI. find
  64. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can You Tell Me How to Get Past Sesame Street? Sentence-… find
  65. Yue Zhao, Xiaolong Jin, Yuanzhuo Wang, and Xueqi Cheng. 2018. Document embedding enhanced event detection with hierarchical and supervised attention. In Proceedings of ACL, pages 414–419. find
  66. Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019b. Adversarial training for weakly supervised event detection. In Proceedings of NAACL, pages 998–1008. find
  67. Xiaozhi Wang, Ziqi Wang, Xu Han, Zhiyuan Liu, Juanzi Li, Peng Li, Maosong Sun, Jie Zhou, and Xiang Ren. 2019c. HMEAE: Hierarchical modular event argument extraction. In Proceedings of EMNLP-IJCNLP, pages 5777–5783. find
  68. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. find our card
  69. Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event detection with multi-order graph convolution and aggregated attention. In Proceedings of EMNLP-IJCNLP, pages 5766–5770. find
  70. Hui Yang, Tat-Seng Chua, Shuguang Wang, and Chun-Keat Koh. 2003. Structured use of external knowledge for event-based open domain question answering. In Proceedings of SIGIR, pages 33–40. find