Elamur Research⌕
☰
home / papers / 2023 / Exploring the Feasibility of ChatGPT for Event Extraction

Exploring the Feasibility of ChatGPT for Event Extraction

Paper · 2023 · Jun Gao, Huan Zhao, Changlong Yu et al.

Summary

The paper asks whether ChatGPT can perform event extraction from a prompt alone, without task-specific training data or fine-tuning. This matters because annotated event data is costly to produce. The authors focus on event detection (finding trigger words and assigning event types) on the ACE 2005 corpus. Their prompts contain a task description, definitions of the event types, and positive and negative demonstrations, and they ask for output as JSON. They compare ChatGPT with the fine-tuned systems Text2Event and EEQA, test which prompt components matter, and look at rare event types and texts containing multiple events. They also run a small usability study with human prompt writers. ChatGPT trails the supervised models consistently. In the long-tail and multi-event settings, it reaches only about half of EEQA's score. Its results also change a lot depending on how the prompt is phrased. The study is an early, cautionary data point for teams considering LLMs as a replacement for trained extractors on schema-heavy information extraction tasks.

Key points

  • On 20 randomly sampled ACE05 test sentences (33 event types), ChatGPT scored 64.00 F1. Text2Event (T5-base) scored 75.76, Text2Event (T5-large) 77.42 and EEQA (BERT-base) 81.82. ChatGPT's main weakness was precision, because it tended to over-predict triggers.
  • In the prompt ablation, removing the event type definitions cost 11.95 F1 and removing the positive example cost 6.17 F1. Removing the negative example actually improved results, possibly because the model treated it as a positive demonstration.
  • ChatGPT almost always produced well-formed structured output: 20/20 samples with the full prompt and 19/20 in some ablated settings.
  • In long-tail and complex (multi-event) scenarios, ChatGPT averaged only 51.04% of EEQA's performance. The paper reports 49.2% on low-frequency events and 56.7% on high-frequency events.
  • In the usability test, four NLP-trained annotators each had five prompt attempts on 10 samples. Scores did not improve steadily across attempts and varied widely between prompt styles, from 26.92 to 66.67 F1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China76 works · 27,918 citations🎓Hong Kong University of Science and TechnologyHong Kong · China58 works · 23,058 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Jun Gao, Huan Zhao, Changlong Yu and Ruifeng Xu. 2023. Exploring the Feasibility of ChatGPT for Event Extraction. arXiv:2303.03836.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.03836 · arXiv 2303.03836

@misc{gao2023exploring,
  title = {{Exploring the Feasibility of ChatGPT for Event Extraction}},
  author = {Jun Gao, Huan Zhao, Changlong Yu and Ruifeng Xu},
  year = {2023},
  doi = {10.48550/arxiv.2303.03836},
  eprint = {2303.03836},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.03836}
}

References

12 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ArXiv, abs/2302.04…
  2. X. Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Conference on Empirical Methods in Natural Language Processing.
  3. Jun Gao, Changlong Yu, Wei Wang, Huan Zhao, and Ruifeng Xu. 2022. Mask-then-fill: A flexible and effective data augmentation framework for event extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4537– 4544, Abu Dhabi, United Arab Emirates. Association for Co… find
  4. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. ArXiv, abs/2301.08745.
  5. Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In Annual Meeting of the Association for Computational Linguistics.
  6. Kang Liu, Yubo Chen, Jian Liu, Xinyu Zuo, and Jun Zhao. 2020. Extracting events and their relations from texts: A survey on recent research progress and challenges. AI Open, 1:22–39. find
  7. Chenwei Lou, Jun Gao, Changlong Yu, Wei Wang, Huan Zhao, Weiwei Tu, and Ruifeng Xu. 2022. Translation-based implicit annotation projection for zero-shot cross-lingual event argument extraction. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retr… find
  8. Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, M. Liao, and Shaoyi Chen. 2021. Text2event: Controllable sequence-to-structure generation for end-to-end event extraction. ArXiv, abs/2106.09232.
  9. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683.
  10. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. ArXiv, abs/2301.08653. — [arXiv:2301.08653](https://arxiv.org/abs/2301.08653) find
  11. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ma… find
  12. Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019. Exploring pre-trained language models for event extraction and generation. In Annual Meeting of the Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2023-exploring-feasibility-chatgpt-event-extraction/