Elamur Research⌕
☰
home / papers / 2021 / Zero-shot Event Extraction via Transfer Learning: Challenges and Insights

Zero-shot Event Extraction via Transfer Learning: Challenges and Insights

Paper · 2021 · Qing Lyu, Hongming Zhang, Elior Sulem et al.

Summary

Event extraction (EE) usually depends on costly supervised annotation tied to a fixed event ontology. This paper tests whether EE can be done with no event training data at all. The authors reuse off-the-shelf pretrained models built for other tasks. Trigger detection is posed as Textual Entailment or Yes/No QA: SRL-based text pieces serve as premises and are checked against one hypothesis per event type. Argument detection is posed as extractive QA, with handcrafted questions for each argument role. The pretrained models are BERT-family models trained on MNLI, BoolQ, QAMR and SQuAD2.0. Design choices were tuned on the ACE-2005 development set, and the resulting system was also applied unchanged to ERE (LDC2015E29). When gold trigger spans are provided, the system beats earlier zero-shot methods on every subtask. It still trails supervised SOTA by a wide margin, and it struggles to locate trigger spans from scratch. A manual error analysis with ablations traces the gap to three sources: weaknesses of the pretrained models, the way the models are queried, and the inherent difficulty of the task. This gives concrete directions for future work on ontology-flexible EE.

Key points

  • The authors describe it as the first system to identify both event triggers and their arguments while using no EE-annotated data for training. It is evaluated on ACE-2005 (7 types / 33 subtypes, split from Lin et al. 2020) and transferred directly to ERE.
  • On ERE with gold trigger spans, trigger classification reaches F1 70.4 on event types shared with ACE but only 19.0 on new, finer-grained types (e.g., MEET vs. CORRESPONDENCE vs. CONTACT).
  • Trigger errors in a 100-example sample:
    • subtle type distinctions: 17%
    • distracting context: 18%
    • insufficient context: 19%
  • Fixes for trigger context problems:
    • Using the full sentence as the premise corrects 59% of insufficient-context errors.
    • A "minimal-pair premise" strategy corrects only 18% of distracting-context errors.
  • Argument errors and ablations:
    • Main error types are competitive entities (24%), ungrammatical questions (15%) and no-answer questions without a competing entity (13%).
    • A QA model trained on QAMR beats one trained on SQuAD2.0 by +16.9 AI and +13.6 AC.
    • An answerable/unanswerable classifier drops from about 86 accuracy on SQuAD to 57 on ACE.
    • Manually fixing question grammar resolves 40% of the related errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States170 works · 60,739 citations🎓Hong Kong University of Science and TechnologyHong Kong · China58 works · 23,058 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Qing Lyu, Hongming Zhang, Elior Sulem and Dan Roth. 2021. Zero-shot Event Extraction via Transfer Learning: Challenges and Insights. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-short.42

@inproceedings{lyu2021zero,
  title = {{Zero-shot Event Extraction via Transfer Learning: Challenges and Insights}},
  author = {Qing Lyu, Hongming Zhang, Elior Sulem and Dan Roth},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.42},
  url = {https://doi.org/10.18653/v1/2021.acl-short.42}
}

References

20 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yunmo Chen, Tongfei Chen, Seth Ebner, Aaron Steven White, and Benjamin Van Durme. 2020. Reading the manual: Event extraction as definition comprehension. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, pages 74–83. find
  2. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671–683.
  5. Hangfeng He, Qiang Ning, and Dan Roth. 2020. Quase: Question-answer driven sentence encoding. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  6. Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018. Zero-Shot Transfer Learning for Event Extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160–2170, Melbourne, Australia. Associat…
  7. Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL-08: Hlt, pages 254–262.
  8. Viet Dac Lai, Thien Huu Nguyen, and Frank Dernoncourt. 2020. Extensively matching for few-shot learning event detection. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 38–45. find
  9. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  10. Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999–8009.
  11. Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event Extraction as Machine Reading Comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1641–1651, Online. Association for Computational Linguistics.
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  13. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Sho…
  14. Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event Detection and Co-reference with Minimal Supervision. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 392–402, Austin, Texas. Association for Computational Linguistics.
  15. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2):257–287.
  16. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789.
  17. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
  18. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  19. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45. find
  20. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lyu-2021-zero-shot-event-extraction-transfer/