Elamur Research⌕
☰
home / papers / 2021 / CLEVE: Contrastive Pre-training for Event Extraction

CLEVE: Contrastive Pre-training for Event Extraction

Paper · 2021 · Ziqi Wang, Xiaozhi Wang, Xu Han et al.

Summary

Event extraction (EE) means locating event triggers, assigning event types, and identifying arguments with their roles. It usually depends on small, costly annotated datasets. PLMs such as BERT help, but their pre-training does not target event knowledge. CLEVE addresses this with contrastive pre-training guided by AMR graphs, which automatic parsers produce from large unlabeled corpora (NYT, English Wikipedia).

The framework has two parts:

  • Text encoder (Transformer): it learns to place words joined by ARG, time or location AMR edges, which are typically trigger–argument pairs, closer together than unrelated words. Negatives are built by substituting other triggers or arguments.
  • Graph encoder (GNN): it is pre-trained with AMR subgraph discrimination to capture transferable event structure.

Before training, multi-word entities are collapsed into single nodes. Using both representations together improves supervised EE on ACE 2005 and MAVEN. The gains are larger in the unsupervised "liberal" setting, where event schemata must be discovered without any annotation. This is valuable wherever event annotation is scarce.

Key points

  • Code and pre-trained checkpoints are released at github.com/THU-KEG/CLEVE.
  • With NYT pre-training, CLEVE reaches 79.8 ED / 61.1 EAE F1 on ACE 2005 and 68.5 ED F1 on MAVEN. With Wikipedia pre-training, the scores are 79.1 / 60.4 and 68.8.
  • On subsets of MAVEN training data, CLEVE's advantage over RoBERTa and BiLSTM+CRF grows as supervision shrinks, which points to usefulness for low-resource EE.
  • Ablations show that structure pre-training matters much more in liberal EE than in supervised EE.
  • Robustness and domain effects:
    • Swapping CAMR for a stronger parser (Xu et al., 2020) gives only modest gains, so CLEVE tolerates parsing errors.
    • Domain match matters: Wikipedia is 2.28× larger than NYT yet does worse on ACE 2005.
    • The in-domain gains come mainly from the semantic component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🏢TencentShenzhen · China66 works · 17,945 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksInformation extraction · 109 works

Cite

Cite as Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li and Jie Zhou. 2021. CLEVE: Contrastive Pre-training for Event Extraction. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.491

@inproceedings{wang2021cleve,
  title = {{CLEVE: Contrastive Pre-training for Event Extraction}},
  author = {Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li and Jie Zhou},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.491},
  url = {https://doi.org/10.18653/v1/2021.acl-long.491}
}

References

69 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacqueline Aguilar, Charley Beller, Paul McNamee, Benjamin Van Durme, Stephanie Strassel, Zhiyi Song, and Joe Ellis. 2014. A comparison of the events and relations across ACE, ERE, TAC-KBP, and FrameNet annotation standards. In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Cor…
  2. David Ahn. 2006. The stages of event extraction. In Proceedings of ACL Workshop on Annotating and Reasoning about Time and Events, pages 1–8.
  3. Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. 2009. A Comparison of Extrinsic Clustering Evaluation Metrics Based on Formal Constraints. Inf. Retr., 12(4):461–486. find
  4. Amit Bagga and Breck Baldwin. 1998. Entity-Based Cross-Document Coreferencing Using the Vector 5 https://creativecommons.org/licenses/ by-sa/3.0/ Space Model. pages 79–85. ↗
  5. In Proceedings of ACL-COLING, Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the Blanks: Distributional Similarity for Relation Learning. In Proceedings of ACL, pages 2895–2905.
  6. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for Sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou…
  7. Nathanael Chambers. 2013. Event Schema Induction with a Probabilistic Entity-Driven Model. In Proceedings of EMNLP, pages 1797–1807. find
  8. Nathanael Chambers and Dan Jurafsky. 2011. Template-Based Information Extraction without the Templates. In Proceedings of ACL-HLT, pages 976–986. find
  9. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of ICML, pages 1597–1607.
  10. Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu, and Jun Zhao. 2017. Automatically Labeled Data Generation for Large Scale Event Extraction. In Proceedings of ACL, pages 409–419.
  11. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of ACL-IJCNLP, pages 167–176.
  12. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In Proceedings of ICLR. find
  13. Wanyun Cui, Guangyu Zheng, and Wei Wang. 2020. Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning. In Proceedings of EMNLP, pages 5511–5520. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  15. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2015. Overview of linguistic resources for the TAC KBP 2015 evaluations: Methodologies and results. In TAC. find
  16. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2016. Overview of Linguistic Resources for the TAC KBP 2016 Evaluations: Methodologies and Results. In TAC. find
  17. Jeremy Getman, Joe Ellis, Zhiyi Song, Jennifer Tracey, and Stephanie Strassel. 2017. Overview of linguistic resources for the tac kbp 2017 evaluations: Methodologies and results. In TAC. find
  18. Goran Glavaš and Jan Šnajder. 2014. Event graphs for information retrieval and multi-document summarization. Expert systems with applications, 41(15):6904–6916. find
  19. Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. Train No Evil: Selective Masking for Task-Guided Pre-Training. In Proceedings of EMNLP, pages 6966–6974. find
  20. Prashant Gupta and Heng Ji. 2009. Predicting Unknown Time Arguments based on Cross-Event Propagation. In Proceedings of ACL-IJCNLP, pages 369–372. find
  21. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL, pages 8342–8360. find
  22. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality Reduction by Learning an Invariant Mapping. In Proceedings of CVPR, volume 2, pages 1735–1742. find
  23. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of CVPR, pages 9726–9735.
  24. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In Proceedings of ICLR. find
  25. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  26. Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, Clare R. Voss, Jiawei Han, and Avirup Sil. 2016. Liberal Event Extraction and Event Schema Induction. In Proceedings of ACL, pages 258–268. find
  27. Lifu Huang and Heng Ji. 2020. Semi-supervised New Event Type Induction and Event Detection. In Proceedings of EMNLP, pages 718–724. find
  28. Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018. Zero-Shot Transfer Learning for Event Extraction. In Proceedings of ACL, pages 2160–2170.
  29. Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020. Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models. In Proceedings of ACL, pages 4859–4870. find
  30. Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL, pages 254–262.
  31. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. In Proceedings of NIPS.
  32. Heng Ji and Ralph Grishman. 2011. Knowledge Base Population: Successful Approaches and Challenges. In Proceedings of ACL, pages 1148–1158. find
  33. Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of EMNLP, pages 3661–3672.
  34. Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning. find
  35. Lingpeng Kong, Cyprien de Masson d’Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. 2020. A Mutual Information Maximization Perspective of Language Representation Learning. In Proceedings of ICLR. find
  36. Viet Dac Lai, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020. Event Detection: Gate Diversity and Syntactic Importance Scores for Graph Convolution Neural Networks. In Proceedings of EMNLP, pages 5405–5411. find
  37. Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of ACL, pages 73–82.
  38. Shasha Liao and Ralph Grishman. 2010. Using document level cross-event inference to improve event extraction. In Proceedings of ACL, pages 789–797.
  39. Amir Pouran Ben Veyseh, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020. Graph Transformer Networks with Syntactic and Semantic Structures for Event Argument Extraction. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3651–3661. find
  40. Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020. GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. In Proceedings of KDD, page 1150–1160. find
  41. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, 6(12):e26752. find
  42. Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui. 2016. Joint Learning Templates and Slots for Event Schema Induction. In Proceedings of NAACL-HLT, pages 428–434. find
  43. Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event Extraction as Machine Reading Comprehension. In Proceedings of EMNLP, pages 1641–1651.
  44. Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018. Jointly Extracting Event Triggers and Arguments by Dependency-Bridge RNN and Tensor-Based Argument Interaction. In Proceedings of AAAI, pages 5916–5923. find
  45. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692. find
  46. Meihan Tong, Bin Xu, Shuai Wang, Yixin Cao, Lei Hou, Juanzi Li, and Jun Xie. 2020. Improving event detection via open-domain trigger knowledge. In Proceedings of ACL, pages 5887–5897. find
  47. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In Proceedings of ICLR.
  48. Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besançon. 2015. Generative Event Schema Induction with Entity Disambiguation. In Proceedings of ACL, pages 188–197. find
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of NIPS, pages 5998– 6008.
  50. Ulrike Von Luxburg. 2007. A tutorial on spectral clustering. Statistics and computing, 17(4):395–416. find
  51. Thien Nguyen and Ralph Grishman. 2018. Graph convolutional networks with argument-aware pooling for event detection. In Proceedings of AAAI, pages 5900–5907. find
  52. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, Relation, and Event Extraction with Contextualized Span Representations. In Proceedings of EMNLP-IJCNLP, pages 5784–5789.
  53. Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of NAACL, pages 300–309.
  54. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  55. ACE 2005 multilingual training corpus. Linguistic Data Consortium, 57. find
  56. Thien Huu Nguyen and Ralph Grishman. 2015. Event Detection and Domain Adaptation with Convolutional Neural Networks. In Proceedings of ACL, pages 365–371.
  57. Chuan Wang, Sameer Pradhan, Xiaoman Pan, Heng Ji, and Nianwen Xue. 2016. CAMR at SemEval-2016 Task 8: An Extended Transition-based AMR Parser. In Proceedings of SemEval, pages 1173–1178.
  58. Chuan Wang, Nianwen Xue, and Sameer Pradhan. 2015. Boosting Transition-based AMR Parsing with Refined Actions and Auxiliary Analyzers. In Proceedings of ACL-IJCNLP, pages 857–862.
  59. Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019a. Adversarial Training for Weakly Supervised Event Detection. In Proceedings of NAACL-HLT, pages 998–1008.
  60. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of EMNLP, pages 1652–1671.
  61. Xiaozhi Wang, Ziqi Wang, Xu Han, Zhiyuan Liu, Juanzi Li, Peng Li, Maosong Sun, Jie Zhou, and Xiang Ren. 2019b. HMEAE: Hierarchical Modular Event Argument Extraction. In Proceedings of EMNLP-IJCNLP, pages 5777–5783. find
  62. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised Feature Learning via Non-parametric Instance Discrimination. In Proceedings of CVPR, pages 3733–3742. find
  63. Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving AMR Parsing with Sequence-to-Sequence Pre-training. In Proceedings of EMNLP, pages 2501–2511. find
  64. Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How Powerful are Graph Neural Networks? In Proceedings of ICLR.
  65. Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event Detection with Multi-Order Graph Convolution and Aggregated Attention. In Proceedings of EMNLP-IJCNLP, pages 5766–5770. find
  66. Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019. Exploring Pre-trained Language Models for Event Extraction and Generation. In Proceedings of ACL, pages 5284–5294.
  67. Yi-Ting Yeh and Yun-Nung Chen. 2019. QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization. In Proceedings of EMNLP-IJCNLP, pages 3370–3375. find
  68. Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. In Proceedings of NeurIPS, pages 5812–5823. find
  69. Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2021-cleve-contrastive-pre-training-event/