Elamur Research⌕
☰
home / papers / 2026 / MUC-4 Revisited: Document-level Event Analysis beyond Span-based Arguments

MUC-4 Revisited: Document-level Event Analysis beyond Span-based Arguments

Summary

Most current event extraction work treats event arguments as contiguous spans copied from the text. This misses information that applications such as conflict research depend on, including actor categories, resolved dates and victim counts. The authors note that MUC-4, a 1992 benchmark on Latin American terrorism usually used for span extraction, originally had richer templates. These include normalised dates, country-level locations, categorical mappings and inferred values, such as weapon type Gun deduced from a mention of bullet wounds. They restore the full original schema and recast MUC-4 as an abstractive, document-level event analysis task. Each field is profiled with domain-openness measures and a new span-match ratio that shows how closely its values align with the source text. They then fine-tune generative decoder-only models (Qwen3 and OLMo-3) with LoRA to output events as JSON. The experiments compare model sizes and post-training stages across span and non-span fields, followed by an error analysis. This reconnects a classic benchmark with recent abstractive datasets such as UCDP-AEC and Lemonade. The provided text omits the result scores.

Key points

  • MUC-4 is reintroduced with all original fields, not just the simplified span-based template used in prior work. That template covered event type, perpetrator individual and organisation, weapon, physical target and victim. The restored schema adds fields such as Date, Country, City, Event Stage, Weapon Type, Perp. Confidence, victim and target numbers, and effects.
  • Event types across all annotated templates are dominated by Attack (56.80%) and Bombing (26.62%), followed by Kidnapping (10.50%).
  • Models are fine-tuned with LoRA via PEFT (r=16, α=16, dropout 0.05), AdamW and a cosine schedule. Tested models include Qwen3 at 0.6B, 4B and larger sizes, plus allenai/OLMo-3-7B-Instruct. Simple instruction prompts were used because chat-style wrapping often produced invalid JSON.
  • Semantic evaluation follows Simon et al. (2025), using Precision@1 and BERTScore, both with bert-base-uncased embeddings for consistency.
  • Code is released at github.com/helenebol/MUC-4-Revisited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2kaxd8nu44bx

@inproceedings{olsen2026muc,
  title = {{MUC-4 Revisited: Document-level Event Analysis beyond Span-based Arguments}},
  author = {Olsen, Helene Bøsei and Velldal, Erik and Øvrelid, Lilja},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2kaxd8nu44bx},
  url = {https://doi.org/10.63317/2kaxd8nu44bx}
}

References

36 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yunmo Chen, William Gantt, Weiwei Gu, Tongfei Chen, Aaron White, and Benjamin Van Durme. 2023. Iterative document-level information extraction via imitation learning. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1858–1874, Dubr… find
  2. Aliva Das, Xinya Du, Barry Wang, Kejian Shi, Jiayuan Gu, Thomas Porter, and Claire Cardie. 2022. Automatic error analysis for document-level information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3960–3975, Du… find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  4. Xinya Du and Claire Cardie. 2020. Document-level event role filler extraction using multi-granularity contextualized encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8010–8020, Online. Association for Computational Linguistics. find
  5. Xinya Du, Alexander Rush, and Claire Cardie. 2021a. Grit: Generative role-filler transformers for document-level event entity extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online. Association for Computational… find
  6. Xinya Du, Alexander Rush, and Claire Cardie. 2021b. Template filling with generative transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 909–914, Online. Association for Computational L… find
  7. Quntian Fang, Feng Liu, Zhen Huang, Zhenliang Guo, Changjian Wang, Dongsheng Li, and Minghao Hu. 2024. Controllable template generation for document-level event extraction. In 2024 4th International Conference on Neural Networks, Information and Communication Engineering (NNICE), pages 325–329. IEEE… find
  8. William Gantt, Alexander Martin, Pavlo Kuchmiichuk, and Aaron Steven White. 2024. Event-keyed summarization. arXiv preprint arXiv:2402.06973. find
  9. Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction. find
  10. I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022. DEGREE: A data-efficient generation-based event extraction model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: H… find our card
  11. Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, and Heng Ji. 2024. TextEE: Benchmark, reevaluation, reflections, and future challenges in event extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12… find
  12. Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. 2021. Document-level entity-based extraction as template generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5257–5269, Online and Punta Cana, Dominican Republic. Association for Computational Ling… find
  13. Ruihong Huang and Ellen Riloff. 2011. Peeling back the layers: Detecting event role fillers in secondary contexts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1137–1147, Portland, Oregon, USA. Association for Computat… find
  14. Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97. find
  15. Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 894–908, Online. Association for Computa… find our card
  16. Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to-structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the… find our card
  17. James Munkres. 1962. Algorithms for the assignment and transportation problems. Society for Industrial and Applied Mathematics, 15:196–210. find
  18. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, R… find our card
  19. Helene Olsen, Étienne Simon, Erik Velldal, and Lilja Øvrelid. 2024. Socio-political events of conflict and unrest: A survey of available datasets. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024), pages 40–53, S… find
  20. Omar Sharif, Joseph Gatto, Madhusudan Basak, and Sarah Masud Preum. 2024. Explicit, implicit, and scattered: Revisiting event extraction to capture complex arguments. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12061–12081, Miami, Florida, USA. As… find
  21. Étienne Simon, Helene Olsen, Huiling You, Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2024. Generative approaches to event extraction: Survey and outlook. In Proceedings of the Workshop on the Future of Event Detection (FuturED), pages 73–86, Miami, Florida, USA. Association for Computational Li… find
  22. Barry Wang, Xinya Du, and Claire Cardie. 2023. Probing representations for document-level event extraction. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12675–12683, Singapore. Association for Computational Linguistics. find
  23. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find our card
  24. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  25. Mengna Zhu, Kaisheng Zeng, JibingWu JibingWu, Lihua Liu, Hongbin Huang, Lei Hou, and Juanzi Li. 2024. LC4EE: LLMs as good corrector for event extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12028–12038, Bangkok, Thailand. Association for Computational Lingui… find
  26. Language Resource References George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ACE) program-tasks, data, and evaluation. In International Conference on Language Resources and Evaluation, volu… find our card
  27. William Gantt, Shabnam Behzad, Hannah An, Yunmo Chen, Aaron White, Benjamin Van Durme, and Mahsa Yarmohammadi. 2024. MultiMUC: Multilingual template filling on MUC-4. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers… find
  28. Clionadh Raleigh, rew Linke, Håvard Hegre, and Joakim Karlsen. 2010. Introducing ACLED: An armed conflict location and event dataset. Journal of Peace Research, 47(5):651–660. find
  29. Sina Semnani, Pingyue Zhang, Wanyue Zhai, Haozhuo Li, Ryan Beauchamp, Trey Billing, Katayoun Kishi, Manling Li, and Monica Lam. 2025. LEMONADE: A large multilingual expert-annotated abstractive event dataset for the real world. In Findings of the Association for Computational Linguistics: ACL 2025,… find
  30. Étienne Simon, Helene Bøsei Olsen, Ramón Carreño, Rahul Mishra, Nikolay Arefyev, Mert Can Yilmaz, Lilja Øvrelid, and Erik Velldal. 2025. Abstractive event analysis of armed conflicts: Introducing the UCDP-AE dataset. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025)… find
  31. Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: annotation of entities, relations, and events. In Proceedings of the the 3rd Workshop on EVENTS: Definition, Detection, Coreference,… find our card
  32. Ralph Sundberg and Erik Melander. 2013. Introducing the UCDP Georeferenced Event Dataset. Journal of Peace Research, 50(4):523–532. find
  33. Beth M. Sundheim. 1992a. Overview of the fourth Message Understanding evaluation and Conference. In Fourth Message Understanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992. find
  34. Beth M. Sundheim. 1992b. Overview of the fourth Message Understanding Evaluation and Conference. In Fourth Message Understanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992. find
  35. Beth M Sundheim. 1996. The message understanding conferences. In TIPSTER TEXT PROGRAM PHASE II: Proceedings of a Workshop held at Vienna, Virginia, May 6-8, 1996, pages 35–37. find
  36. MeiHan Tong, Bin Xu, Shuai Wang, Meihuan Han, Yixin Cao, Jiangqi Zhu, Siyu Chen, Lei Hou, and Juanzi Li. 2022. DocEE: A large-scale and fine-grained benchmark for document-level event extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/olsen-2026-muc-4-revisited-document-level/