Elamur Research⌕
☰
home / papers / 2026 / Evaluation Pitfalls and Challenges in Multimedia Event Extraction

Evaluation Pitfalls and Challenges in Multimedia Event Extraction

Paper · 2026 · Philipp Seeberger, Steffen Freisinger, Tobias Bocklet et al.

Summary

Multimedia event extraction (MEE) identifies events and their arguments jointly across text and images. It also links mentions of the same real-world event across modalities through cross-modal coreference. Reported gains on this task keep growing, but the authors question whether these numbers are comparable or realistic. They audit the M2E2 benchmark and 15 peer-reviewed MEE studies evaluated on it. The audit groups the problems into three families: diverging data preprocessing, mismatched task assumptions, and lenient scoring setups. Examples of these problems include trigger postprocessing, verb refinement, evaluating only a filtered test subset, and relying on gold image–text coreference.

To address these problems, they release StrictEval, an evaluation framework with tighter constraints. They then run controlled experiments and re-evaluate several recent systems under both original and strict settings. Small, often unreported evaluation choices turn out to shift scores dramatically, with multimedia results suffering most. Cross-modal coreference emerges as the main unsolved bottleneck. The work matters because it shows that apparent progress in MEE may be largely an artifact of evaluation. It also urges the field toward standardized, transparent protocols.

Key points

  • The study uses M2E2, which is built from Voice of America news. It contains 245 documents, 6,167 sentences and 1,014 images, covering 8 event types and 15 argument roles. Its annotations include 1,297 textual events, 391 visual events and 309 multimedia events. There is no training split, so models are trained on ACE (text) and on imSitu with SWiG groundings (images).
  • The literature survey covers 2020–2025 and found 18 articles. Three were excluded because they used custom splits or new metrics. The authors locate nine pitfall points (P1–P9) along the pipeline.
  • Restricting evaluation to a test subset with prior knowledge inflates threshold-based methods by up to +21.2 F1. It can also reverse method rankings: on the filtered subset threshold-based methods score 30.9 F1 versus 20.4 for bipartite matching, although bipartite matching performs best under realistic conditions.
  • Four systems were re-assessed with StrictEval: SSGPF (Chen et al. 2025), the model of Cao et al. 2025, the 2024 system by Seeberger et al., and the 2023 approach by Du et al. These systems lose up to 56.8 F1 on multimedia event detection and up to 33.6 F1 on multimedia argument extraction.
  • The code is released as StrictEval at github.com/seebergerph/StrictEval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georg Simon Ohm University of Applied Sciences NurembergNuremberg · Germany3 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksInformation extraction · 167 works

Cite

Cite as Philipp Seeberger, Steffen Freisinger, Tobias Bocklet and Korbinian Riedhammer. 2026. Evaluation Pitfalls and Challenges in Multimedia Event Extraction. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1426

@inproceedings{seeberger2026evaluation,
  title = {{Evaluation Pitfalls and Challenges in Multimedia Event Extraction}},
  author = {Philipp Seeberger, Steffen Freisinger, Tobias Bocklet and Korbinian Riedhammer},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1426},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1426}
}

References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events, pages 1–8, Sydney, Australia. Association for Computational Linguistics.
  2. Jianwei Cao, Yanli Hu, Zhen Tan, and Xiang Zhao. 2025. Cross-modal multi-task learning for multimedia event extraction. Proceedings of the AAAI Conference on Artificial Intelligence, 39(11):11454– 11462. find
  3. Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, and Shih-Fu Chang. 2021. Joint multimedia event extraction from video and article. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 74–88, Punta Cana, Dominican Republic. Assoc… find
  4. Xinrong Chen, Xiang Yuan, Haochen Li, Hang Yang, Guanyu Wang, Weiping Li, and Tong Mo. 2025. Stepwise schema-guided prompting framework with parameter efficient instruction tuning for multimedia event extraction. In 2025 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Zilin Du, Yunxin Li, Xu Guo, Yidan Sun, and Boyang Li. 2023. Training multimedia event extraction with generated images and captions. In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, page 5504–5513, New York, NY, USA. Association for Computing Machinery. find
  7. Maofu Liu, Zhenyi Hu, Bingying Zhou, Huijun Hu, Chen Qiu, and Xiaokang Zhang. 2025a. Cross-modal event extraction based on adaptive feature selection and semantic-aware graph. Knowledge-Based Systems, 326:114038. find
  8. Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, and Heng Ji. 2024. TextEE: Benchmark, reevaluation, reflections, and future challenges in event extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12… find
  9. H. W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97. find
  10. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020. The Open Images Dataset V4: Unified Image Classification, Object Detection, and Visual Relationship D… find
  11. Jiaqi Li, Chuanyi Zhang, Miaozeng Du, Dehai Min, Yongrui Chen, and Guilin Qi. 2023. Three stream based multi-level event contrastive learning for text-video event extraction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1666–1676, Singapore. Associ… find
  12. Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022. Clip-event: Connecting text and images with event structures. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16399–16408. find
  13. Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2557–2568, Online. Association for… find
  14. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham. Springer International Publishing.
  15. Jian Liu, Yufeng Chen, and Jinan Xu. 2022. Multimedia event extraction from news with a unified contrastive Maofu Liu, Bingying Zhou, Huijun Hu, Chen Qiu, and Xiaokang Zhang. 2025b. Cross-modal event extraction via visual event grounding and semantic relation filling. Information Processing Manageme… find
  16. Yang Liu, Fang Liu, Licheng Jiao, Qianyue Bao, Long Sun, Shuo Li, Lingling Li, and Xu Liu. 2024. Multi-grained gradual inference model for multimedia event extraction. IEEE Transactions on Circuits and Systems for Video Technology, 34(10):10507–10520. find
  17. Farhad Moghimifar, Fatemeh Shiri, Van Nguyen, Yuan-Fang Li, and Gholamreza Haffari. 2023. Theia: Weakly supervised multimodal event extraction from incomplete data. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Cha… find
  18. Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Arthur Baker, Rahul Ghosh, James H. Martin, Nathaniel Blanchard, and Nikhil Krishnaswamy. 2024. Multimodal cross-document event coreference resolution using linear semantic transfer and mixed-modality ensembles. In Proceedings of the 2024 Joi…
  19. Hao Peng, Xiaozhi Wang, Feng Yao, Zimu Wang, Chuzhao Zhu, Kaisheng Zeng, Lei Hou, and Juanzi Li. 2023a. OmniEvent: A comprehensive, fair, and easy-to-use toolkit for event understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,… find
  20. Hao Peng, Xiaozhi Wang, Feng Yao, Kaisheng Zeng, Lei Hou, Juanzi Li, Zhiyuan Liu, and Weixing Shen. 2023b. The devil is in the details: On the pitfalls of event extraction evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9206–9227, Toronto, Canada. Associatio… find
  21. Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded Situation Recognition. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, volume 12349, pages 314–332. Springer International Publishing, Cham. Series T… find
  22. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint. Version Number: 1.
  23. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788. find
  24. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  25. Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. 2021. Visual semantic role labeling for video understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  26. Kate Sanders, Reno Kriz, David Etter, Hannah Recknor, Alexander Martin, Cameron Carpenter, Jingyang Lin, and Benjamin Van Durme. 2024. Grounding partially-defined events in multimodal data. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15905–15927, Miami, Florida, U… find
  27. Philipp Seeberger, Dominik Wagner, and Korbinian Riedhammer. 2024. MMUTF: Multimodal multimedia event argument extraction with unified template filling. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6539–6548, Miami, Florida, USA. Association for Computational Lingu… find
  28. Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: Annotation of entities, relations, and events. In Proceedings of the 3rd Workshop on EVENTS: Definition, Detection, Coreference, and…
  29. Lin Sun, Kai Zhang, Qingyuan Li, and Renze Lou. 2024. Umie: Unified multimodal information extraction with instruction tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19062– 19070. find
  30. Yuxuan Sun, Kai Zhang, and Yu Su. 2023. Multimodal question answering for unified information extraction. Preprint, arXiv:2310.03017. — [arXiv:2310.03017](https://arxiv.org/abs/2310.03017) find
  31. Meihan Tong, Shuai Wang, Yixin Cao, Bin Xu, Juanzi Li, Lei Hou, and Tat-Seng Chua. 2020. Image enhanced event detection in news articles. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9040–9047. find
  32. Rejin Varghese and Sambath M. 2024. Yolov8: A novel object detection algorithm with enhanced performance and robustness. In 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), pages 1–6. find
  33. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
  34. Walker, Christopher, Strassel, Stephanie, Medero, Julie, and Maeda, Kazuaki. 2006. find
  35. ACE 2005 Multilingual Training Corpus. Artwork Size: 1572864 KB Pages: 1572864 KB. find
  36. Bin Wang, Meishan Zhang, Hao Fei, Yu Zhao, Bobo Li, Shengqiong Wu, Wei Ji, and Min Zhang. 2024. Speechee: A novel benchmark for speech event extraction. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 10449–10458, New York, NY, USA. Association for Computing Machi… find
  37. Shuo Wang, Meizhi Ju, Yunyan Zhang, Yefeng Zheng, Meng Wang, and Guilin Qi. 2023. Cross-modal contrastive learning for event extraction. In Database Systems for Advanced Applications, pages 699–715, Cham. Springer Nature Switzerland. find
  38. Xiaoyu Wang, Tao Sun, Gengchen Liu, Zhi Yang, Jiahui Liu, and Zimeng Xu. 2025. Mgfsg-ee: A method based on multi-grained fusion and scene graph enhancement for event extraction. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, page 3103–3112,… find
  39. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652– 1671, Onli…
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  41. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
  42. Fuyu Xing, Zimu Wang, Wei Wang, and Haiyang Zhang. 2025. Benchmarking and improving LVLMs on event extraction from multimedia documents. In Proceedings of the 18th International Natural Language Generation Conference, pages 734–742, Hanoi, Vietnam. Association for Computational Linguistics. find
  43. Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Conference on Computer Vision and Pattern Recognition. find
  44. Jiaao Yu, Yijing Lin, Zhipeng Gao, Xuesong Qiu, and Lanlan Rui. 2025. Multimedia event extraction with LLM knowledge editing. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4116–4124, Suzhou, China. Association for Computational Linguistics. find
  45. Li Yuan, Yi Cai, Xudong Shen, Qing Li, Qingbao Huang, Zikun Deng, and Tao Wang. 2025. Collaborative multi-lora experts with achievement-based multi-tasks loss for unified multimodal information extraction. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence,… find
  46. Meishan Zhang, Hao Fei, Bin Wang, Shengqiong Wu, Yixin Cao, Fei Li, and Min Zhang. 2024. Recognizing everything from all modalities at once: Grounded multimodal universal information extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14498–14511, Bangkok, Thail… find
  47. Tongtao Zhang, Spencer Whitehead, Hanwang Zhang, Hongzhi Li, Joseph Ellis, Lifu Huang, Wei Liu, Heng Ji, and Shih-Fu Chang. 2017. Improving event extraction via multimodal integration. In Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, page 270–278, New York, NY, USA. Ass… find
  48. Yuhui Zhang, Yongxiu Xu, Minghao Tang, Xinkui Lin, Yubin Wang, Hongbo Xu, and Gaopeng Gou. 2025. Rda: Regularized domain adaptation for multimedia event extraction. In Advanced Intelligent Computing Technology and Applications, pages 308–319, Singapore. Springer Nature Singapore. find
  49. Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2021. Revisiting the evaluation of end-to-end event extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4609– 4617, Online. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seeberger-2026-evaluation-pitfalls-challenges-multimedia-event/