Elamur Research⌕
☰
home / papers / 2022 / MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

Summary

Event relation extraction (ERE) covers coreference, temporal, causal and subevent links between events. Progress has been limited by datasets that are small (hundreds of documents) and that label only one or two relation types at a time. This prevents models from exploiting how relations constrain each other; for example, a cause must precede its effect. The authors build MAVEN-ERE on top of the MAVEN event detection corpus: 4,480 English Wikipedia documents, crowd-annotated for all four relation types in the same texts. To keep costs manageable, they adapt the O'Gorman et al. (2016) guidelines into a staged pipeline. Coreference is annotated first. Temporal relations come next, using a new timeline scheme in which annotators place events on time axes rather than labelling every pair. Causal and subevent links are added last, narrowed down by temporal constraints and transitivity. The resulting resource is at least ten times larger than earlier datasets for every ERE task. RoBERTa-based baselines show the tasks remain difficult, that more training data helps, and that simple joint training across relation types improves scores.

Key points

  • Size: 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations and 15,841 subevent relations. It is described as the first million-scale human-annotated ERE dataset.
  • Temporal annotation uses 6 relation types (BEFORE, CONTAINS, OVERLAP, BEGINS-ON, ENDS-ON, SIMULTANEOUS) plus ISO-TimeML TIMEXs (DATE, TIME, DURATION, PREPOSTEXP). Per 100 words, it yields over 6× the temporal relations of MATRES.
  • Quality: each document was labelled by 3 annotators with majority voting. Pairwise B-Cubed F-1 for coreference averages 91%, and Fleiss' kappa for TIMEX annotation is 78.4%.
  • Error analysis shows that most mistakes come from deciding whether a relation exists at all. Long-range dependencies remain a challenge.
  • Limitations: English only and within-document relations only. Data and code are available at github.com/THU-KEG/MAVEN-ERE (built on MAVEN, which is licensed under CC BY-SA 4.0).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

56 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 55 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.60 · arXiv 2211.07342

@inproceedings{wang2022maven,
  title = {{MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction}},
  author = {Wang, Xiaozhi and Chen, Yulin and Ding, Ning and Peng, Hao and Wang, Zimu and Lin, Yankai and Han, Xu and Hou, Lei and Li, Juanzi and Liu, Zhiyuan and Li, Peng and Zhou, Jie},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.60},
  eprint = {2211.07342},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.60}
}

References

76 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Aldawsari and Mark Finlayson. 2019. Detecting subevents using discourse and narrative features. In Proceedings of ACL, pages 4780–4790. find our card
  2. James F. Allen. 1983. Maintaining knowledge about temporal intervals. Commun. ACM, 26(11):832–843. find
  3. Amit Bagga and Breck Baldwin. 1998. Entity-Based Cross-Document Coreferencing Using the Vector Space Model. In Proceedings of ACL-COLING, pages 79–85. find
  4. Cosmin Bejan and Sanda Harabagiu. 2008. A linguistic resource for discovering event structures and resolving event coreference. In Proceedings of LREC. find our card
  5. Pengfei Cao, Xinyu Zuo, Yubo Chen, Kang Liu, Jun Zhao, Yuguang Chen, and Weihua Peng. 2021. Knowledge-enriched event causality identification via latent structure induction networks. In Proceedings of ACL-IJCNLP, pages 4862–4872. find
  6. Tommaso Caselli and Piek Vossen. 2017. The event StoryLine corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of the Events and Stories in the News Workshop, pages 77– 86. find
  7. Nathanael Chambers. 2013. Event Schema Induction with a Probabilistic Entity-Driven Model. In Proceedings of EMNLP, pages 1797–1807. find
  8. Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics, 2:273– 284. find
  9. Nathanael Chambers and Daniel Jurafsky. 2008. Jointly combining implicit constraints improves temporal ordering. In Proceedings of EMNLP, pages 698–706. find
  10. Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of EMNLP, pages 1603–1614. find
  11. Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of EMNLP, pages 2124–2133. find our card
  12. Prafulla Kumar Choubey and Ruihong Huang. 2018. Improving event coreference resolution by modeling correlations between event coreference chains and document topic structures. In Proceedings of ACL, pages 485–495. find our card
  13. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of LREC, pages 4545–4552. find our card
  14. Pascal Denis and Philippe Muller. 2011. Predicting globally-coherent temporal structures from texts via endpoint inference and graph decomposition. In Proceedings of IJCAI, page 1788–1793. find
  15. Dmitriy Dligach, Timothy Miller, Chen Lin, Steven Bethard, and Guergana Savova. 2017. Neural temporal relation extraction. In Proceedings of EACL, pages 746–751. find
  16. Quang Do, Yee Seng Chan, and Dan Roth. 2011. Minimally supervised event causality identification. In Proceedings of EMNLP, pages 294–303. find
  17. Jesse Dunietz, Lori Levin, and Jaime Carbonell. 2017. The BECauSE corpus 2.0: Annotating causality and overlapping relations. In Proceedings of the 11th Linguistic Annotation Workshop, pages 95–104. find
  18. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2015. Overview of linguistic resources for the TAC KBP 2015 evaluations: Methodologies and results. In TAC. find
  19. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2016. Overview of Linguistic Resources for the TAC KBP 2016 Evaluations: Methodologies and Results. In TAC. find
  20. Lei Gao, Prafulla Kumar Choubey, and Ruihong Huang. 2019. Modeling document-level causal structures for event causal relation identification. In Proceedings of NAACL-HLT, pages 1808–1817. find
  21. Alfonso Gerevini and Lenhart Schubert. 1995. Efficient algorithms for qualitative reasoning about time. Artificial Intelligence, 74(2):207–248. find
  22. Jeremy Getman, Joe Ellis, Zhiyi Song, Jennifer Tracey, and Stephanie Strassel. 2017. Overview of linguistic resources for the tac kbp 2017 evaluations: Methodologies and results. In TAC. find
  23. Goran Glavaš, Jan Šnajder, Marie-Francine Moens, and Parisa Kordjamshidi. 2014. HiEve: A corpus for extracting event hierarchies from news stories. In Proceedings of LREC, pages 3678–3683. find
  24. Ralph Grishman and Beth Sundheim. 1996. Message understanding conference- 6: A brief history. In Proceedings of COLING. find
  25. Chikara Hashimoto, Kentaro Torisawa, Julien Kloetzer, Motoki Sano, István Varga, Jong-Hoon Oh, and Yutaka Kidawara. 2014. Toward future scenario generation: Extracting event causality exploiting semantic relation, context, and association features. In Proceedings of ACL, pages 987–997. find
  26. Yu Hong, Tongtao Zhang, Tim O’Gorman, Sharone Horowit-Hendler, Heng Ji, and Martha Palmer. 2016. Building a cross-document event-event relation corpus. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 1–6. find
  27. Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In Proceedings of Workshop on Events: Definition, Detection, Coreference, and Representation, pages 21– 28. find
  28. Rei Ikuta, Will Styler, Mariah Hamang, Tim O’Gorman, and Martha Palmer. 2014. Challenges of adding causation to richer event descriptions. In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 12–20. find
  29. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of EMNLP-IJCNLP, pages 5803–5808. find our card
  30. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2018. Question answering as global reasoning over semantic abstractions. In Proceedings of AAAI. find
  31. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  32. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of EMNLP-CoNLL, pages 489–500. find our card
  33. Willem JM Levelt. 1993. Speaking: From intention to articulation. MIT press. find
  34. Jian Liu, Yubo Chen, and Jun Zhao. 2020a. Knowledge enhanced event causality identification with mention masking generalizations. In Proceedings of IJCAI, pages 3608–3614. find
  35. Kang Liu, Yubo Chen, Jian Liu, Xinyu Zuo, and Jun Zhao. 2020b. Extracting events and their relations from texts: A survey on recent research progress and challenges. AI Open, 1:22–39. find
  36. Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018a. Joint reasoning for temporal and causal relations. In Proceedings of ACL, pages 2278–2288. find
  37. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692. find
  38. Zhengzhong Liu, Jun Araki, Eduard Hovy, and Teruko Mitamura. 2014. Supervised within-document event coreference using information propagation. In Proceedings of LREC, pages 4539–4544. find our card
  39. Jing Lu and Vincent Ng. 2021a. Constrained multi-task learning for event coreference resolution. In Proceedings of NAACL-HLT, pages 4504–4514. find our card
  40. Jing Lu and Vincent Ng. 2021b. Conundrums in event coreference resolution: Making sense of the state of the art. In Proceedings of EMNLP, pages 1368–1380. find our card
  41. Yaojie Lu, Hongyu Lin, Jialong Tang, Xianpei Han, and Le Sun. 2022. End-to-end neural event coreference resolution. Artificial Intelligence, 303:103632. find
  42. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of EMNLP-HLT, pages 25–32. find our card
  43. George A Miller and Philip N Johnson-Laird. 2013. Language and perception. In Language and Perception. Harvard University Press. find
  44. Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begoña Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the News-Reader multilingual event and time corpus. In Proceedings of LREC, pages 4417–4422. find
  45. Paramita Mirza, Rachele Sprugnoli, Sara Tonelli, and Manuela Speranza. 2014. Annotating causality in the TempEval-3 corpus. In Proceedings of the EACL 2014 Workshop on Computational Approaches to Causality in Language (CAtoCL), pages 10–19. find
  46. Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Vanderwende. 2016. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. In Proceedings of the Fourth Workshop on Events, pages 51–61. find
  47. Aakanksha Naik, Luke Breitfeller, and Carolyn Rose. 2019. TDDiscourse: A dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 239–249. find
  48. Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In Proceedings of EMNLP, pages 1027–1037. find
  49. Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019. An improved neural baseline for temporal relation extraction. In Proceedings of EMNLP-IJCNLP, pages 6203–6209. find
  50. Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi-axis annotation scheme for event temporal relations. In Proceedings of ACL, pages 1318–1328. find
  51. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of CNS, pages 47–56. find our card
  52. Steven Pinker. 2013. Learnability and Cognition, new edition: The Acquisition of Argument Structure. MIT press. find
  53. Sameer S. Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Identifying entities and events in ontonotes. In Proceedings of ICSC, pages 446–453. find
  54. James Pustejovsky, José M Castano, Robert Ingria, Roser Sauri, Robert J Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir R Radev. 2003a. Timeml: Robust specification of event and temporal expressions in text. New directions in question answering, 3:28–34. find
  55. James Pustejovsky, Patrick Hanks, Roser Saurí, Andrew See, Rob Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. 2003b. The TimeBank corpus. Proceedings of Corpus Linguistics. find
  56. James Pustejovsky, Kiyong Lee, Harry Bunt, and Laurent Romary. 2010. ISO-TimeML: An international standard for semantic annotation. In Proceedings of LREC. find
  57. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2Mind: Commonsense inference on events, intents, and reactions. In Proceedings of ACL, pages 463–473. find
  58. M. Recasens and E. Hovy. 2011. Blanc: Implementing the rand index for coreference evaluation. Nat. Lang. Eng., 17(4):485–510. find
  59. Nils Reimers, Nazanin Dehghani, and Iryna Gurevych. 2016. Temporal anchoring of events for the Time-Bank corpus. In Proceedings of ACL, pages 2195– 2204. find
  60. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: an atlas of machine commonsense for if-then reasoning. In Proceedings of AAAI, pages 3027– 3035. find
  61. William F. Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014. Temporal annotation in the clinical domain. Transactions of the Association for Computational Linguistics, 2:143– 1… find
  62. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP: System Demonstrations, pages… find our card
  63. Fiona Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk, Tadashi Nomoto, Hansi Het-tiarachchi, Iqra Ameer, Onur Uca, Farhana Ferdousi Liza, and Tiancheng Hu. 2022. The causal news corpus: Annotating causal relations in event sentences from news. In Proceedings of LREC, pages 2298– 231… find
  64. Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. 2020. Transomcs: From linguistic graphs to commonsense knowledge. In Proceedings of IJ-CAI, pages 4004–4010. find
  65. Xingwei Tan, Gabriele Pergola, and Yulan He. 2021. Extracting event temporal relations via hyperbolic geometry. In Proceedings of EMNLP, pages 8065– 8077. find
  66. Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In Proceedings of ACL, pages 7579–7589. find
  67. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In Proceedings of SemEval, pages 1–9. find
  68. Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Jessica Moszkowicz, and James Pustejovsky. 2009. The tempeval challenge: identifying temporal relations in text. Language Resources and Evaluation, 43(2):161–179. find
  69. Marc Verhagen, Roser Saurí, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 task 13: TempEval-2. In Proceedings of SemEval, pages 57– 62. find
  70. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC. find our card
  71. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  72. ACE 2005 multilingual training corpus. Linguistic Data Consortium, 57. find
  73. Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020a. Joint constrained learning for event-event relation extraction. In Proceedings of EMNLP, pages 696–706. find
  74. Haoyu Wang, Hongming Zhang, Muhao Chen, and Dan Roth. 2021. Learning constraints and descriptive segmentation for subevent detection. In Proceedings of EMNLP, pages 5216–5226. find
  75. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020b. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of EMNLP, pages 1652–1671. find our card
  76. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/wang-2022-maven-ere-unified-large-scale/