Elamur Research⌕
☰
home / papers / 2025 / Large Language Model-Based Event Relation Extraction with Rationales

Large Language Model-Based Event Relation Extraction with Rationales

Paper · 2025 · Zhilei Hu, Zixuan Li, Xiaolong Jin et al.

Summary

The paper addresses document-level Event Relation Extraction (ERE), which covers temporal, causal, subevent and coreference links, using fine-tuned LLMs. Existing LLM approaches have three weaknesses. Classifying every event pair is slow. Output length limits cause relations to be missed in event-dense documents. Training also ignores the reasoning behind each label.

The proposed method, LLMERE, recasts ERE as a multi-answer question: given a target event, the model lists all related events and their relation subtypes in a single pass. Each relation type is treated as its own subtask with a dedicated task description. To keep outputs short, the full document is copied several times, and each copy marks only a random, evenly spread subset of candidate events. The predictions from these copies are then merged. The model is also trained to produce rationales alongside its answers, namely coreference information and transitive relation chains that follow logical rules.

On three standard benchmarks, LLMERE outperforms state-of-the-art baselines in F1. This shows that LLMs can handle long, event-rich documents efficiently.

Key points

  • Answering per target event instead of per pair lowers training and inference complexity from O(n²) to O(n).
  • The partitioning strategy creates m = ⌈(n−1)/k⌉ copies of the document, where k caps the number of highlighted events per copy. Events are marked with numbered angle-bracket tags, such as <e0 Hurricane>.
  • The datasets are:
    • MAVEN-ERE: the original training set is split 8:2 into train/validation, and the original validation set serves as the test set.
    • MATRES: 182/73/20 documents.
    • HiEve: 100 documents split 60/20/20.
  • The subtypes include six temporal ones (BEFORE, OVERLAP, CONTAINS, SIMULTANEOUS, ENDS-ON, BEGINS-ON), two causal ones (CAUSE, PRECONDITION), SUBEVENT and COREFERENCE.
  • The code is released at github.com/HerbertHu/LLMERE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInformation extraction · 167 works

Cite

Cite as Zhilei Hu, Zixuan Li, Xiaolong Jin, Long Bai, Jiafeng Guo and Xueqi Cheng. 2025. Large Language Model-Based Event Relation Extraction with Rationales. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hu2025large,
  title = {{Large Language Model-Based Event Relation Extraction with Rationales}},
  author = {Zhilei Hu, Zixuan Li, Xiaolong Jin, Long Bai, Jiafeng Guo and Xueqi Cheng},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.500/}
}

References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In this paper, we proposed an LLM-based method References with rationales for ERE (LLMERE). It utilized a question-and-answer format that enumerates mul- Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and tiple answers to reduce the task complexity from Jimmy Lin. 2019. Docbert: Bert for document cla… find
  2. Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics.
  3. Long Bai, Saiping Guan, Jiafeng Guo, Zixuan Li, Xiaolong Jin, and Xueqi Cheng. 2021. Integrating deep event-level and script-level information for script event prediction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9869–9878, Online and Punta Can…
  4. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),…
  5. Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603–1614, Copenhagen, Denmark. Association for Computational Linguistics. find
  6. Meiqi Chen, Yixin Cao, Kunquan Deng, Mukai Li, Kun Wang, Jing Shao, and Yan Zhang. 2022. Ergo: Event relational graph transformer for document-level event causality identification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2118–2128, Gyeongju, Republic o… find
  7. Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, and Dongsheng Li. 2024. Improving large language models in event relation logical prediction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9451–9478, Bangkok, Thailand… find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  9. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation…
  10. Chuang Fan, Daoxing Liu, Libo Qin, Yue Zhang, and Ruifeng Xu. 2022. Towards event-level causal relation identification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, pages 1828–1833, New York, NY, USA. Association for C… find
  11. Goran Glavaš, Jan Šnajder, Parisa Kordjamshidi, and Marie-Francine Moens. 2014. Hieve: A corpus for extracting event hierarchies from news stories. In Proceedings of 9th Language Resources and Evaluation Conference, pages 3678–3683. ELRA. find
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  13. Yong Guan, Xiaozhi Wang, Lei Hou, Juanzi Li, Jeff Z. Pan, Jiaoyan Chen, and Freddy Lecue. 2024. Tacoere: Cluster-aware compression for event relation extraction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 202… find
  14. Rujun Han, I. Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019. Deep structured neural network for event temporal relation extraction. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 666–106, Stroudsburg. ACL Press. find
  15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  16. Zhilei Hu, Zixuan Li, Daozhu Xu, Long Bai, Cheng Jin, Xiaolong Jin, Jiafeng Guo, and Xueqi Cheng. 2023. Protoem: A prototype-enhanced matching framework for event relation extraction. Preprint, arXiv:2309.12892. — [arXiv:2309.12892](https://arxiv.org/abs/2309.12892) find
  17. Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1340– 1350, Florence, Italy. Association f…
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  19. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
  20. Mehwish Riaz and Roxana Girju. 2013. Toward a better understanding of causality between verbal events: Extraction and analysis of the causal power of verb-verb associations. In Proceedings of the SIGDIAL 2013 Conference, pages 21–30, Metz, France. Association for Computational Linguistics. find
  21. Hieu Man, Nghia Trung Ngo, Linh Ngo Van, and Thien Huu Nguyen. 2022. Selecting optimal context sentences for event-event relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11058–11066. find
  22. Mehwish Riaz and Roxana Girju. 2014. Recognizing causality in verb-noun pairs via noun and verb semantics. In Proceedings of the EACL 2014 Workshop on Computational Approaches to Causality in Language (CAtoCL), pages 48–57, Gothenburg, Sweden. Association for Computational Linguistics. find
  23. Puneet Mathur, Rajiv Jain, Franck Dernoncourt, Vlad Morariu, Quan Hung Tran, and Dinesh Manocha. 2021. Timers: Document-level temporal relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura… find
  24. Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019. An improved neural baseline for temporal relation extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages… find
  25. Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi-axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1318–1328, Melbourne, Australia. Association for Computational Linguistics. find
  26. Jong-Hoon Oh, Kentaro Torisawa, Canasai Kruengkrai, Ryu Iida, and Julien Kloetzer. 2017. Multi-column convolutional neural networks with causality-attention for why-question answering. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM ’17, pages 415–424, Ne… find
  27. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. find
  28. Minh Tran Phu and Thien Huu Nguyen. 2021. Graph convolutional networks for event causality identification with rich document-level structures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3480… find
  29. M. Recasens and E. Hovy. 2011. Blanc: Implementing the rand index for coreference evaluation. Natural Language Engineering, 17(4):485–510. find
  30. Hugo Touvron, Louis Martin, Kevin Stone, and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  31. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995.
  32. Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. 2022. Maven-ere: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. In Proceedings of the 2022 Conference on Emp…
  33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837.
  34. Kangda Wei, Aayush Gautam, and Ruihong Huang. 2024. Are llms good annotators for discourse-level event relation extraction? Preprint, arXiv:2407.19568. find
  35. Haoyang Wen and Heng Ji. 2021. Utilizing relative event time to enhance event-event temporal relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10431–10437, Online and Punta Cana, Dominican Republic. Association for Computational Lin… find
  36. Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2021. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10266–10284, Online and Punta Cana, Dominican Republic. Association for Computatio…
  37. Wang Xu, Kehai Chen, Lili Mou, and Tiejun Zhao. 2022. Document-level relation extraction with sentences importance estimation and focusing. In Proceedings If Relation(A, B) ∧ Relation(B, C) of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  38. BEFORE ∧ BEFORE BEFORE ∧ OVERLAP BEFORE ∧ CONTAINS BEFORE ∧ SIMULTANEOUS BEFORE ∧ ENDS-ON BEFORE ∧ BEGINS-ON OVERLAP ∧ BEFORE OVERLAP ∧ SIMULTANEOUS CONTAINS ∧ CONTAINS CONTAINS ∧ SIMULTANEOUS SIMULTANEOUS ∧ BEFORE SIMULTANEOUS ∧ OVERLAP SIMULTANEOUS ∧ CONTAINS SIMULTANEOUS ∧ SIMULTANEOUS SIMULTANEO… find
  39. Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023. Zero-shot temporal relation extraction with chatgpt. Preprint, arXiv:2304.05454. find
  40. Baiyan Zhang, Qin Chen, Jie Zhou, Jian Jin, and Liang He. 2024. Enhancing event causality identification with rationale and structure-aware causal question answering. Preprint, arXiv:2403.11129. — [arXiv:2403.11129](https://arxiv.org/abs/2403.11129) find
  41. Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangko…
  42. Jie Zhou, Shenpo Dong, Hongkui Tu, Xiaodong Wang, and Yong Dou. 2022. Rsgt: Relational structure guided temporal relation extraction. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2001–2010, Gyeongju, Republic of Korea. International Committee on Computation… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2025-large-language-model-based-event/