Elamur Research⌕
☰
home / papers / 2026 / MEUR: A Benchmark for Evaluating Vision-Language Models on Multimodal Event Understanding and Reasoning

MEUR: A Benchmark for Evaluating Vision-Language Models on Multimodal Event Understanding and Reasoning

Summary

Most visual question answering benchmarks test objects, attributes and spatial relations. Datasets such as VQA-E and GQA include few questions about actions or events, and those can often be solved through shortcuts. The authors introduce MEUR, a benchmark that checks whether VLMs understand what is happening in an image, who takes part, and how participants relate to the depicted event. It holds 1,200 images and 4,217 open-ended questions across five strategies, ranging from naming an event to counting, comparison and spotting questions with no valid answer.

The questions come from a semi-automated pipeline. Images and verb annotations are drawn from SwiG, with event types and roles aligned to the MEED schema. GPT-4o mini drafts the simpler questions, GPT-4o filters them, and humans verify these and write the harder reasoning items. GPT-4o mini then rephrases the human-written items for variety.

Tests on non-thinking and thinking VLMs show the task is hard: the strongest system reaches only about 41% accuracy. MEUR exposes event-specific weaknesses and hallucinations that entity-focused datasets overlook.

Key points

  • Five question strategies:
    • Event Recognition and Argument Identification mirror the textual tasks of event detection and event argument extraction.
    • Argument Associate Identification and Counting and Comparison target reasoning.
    • Unanswerable Questions probe hallucination.
  • Pipeline sources: the "VLM-then-Human" pipeline reuses SwiG (Pratt et al., 2020) and the MEED event schema (Wang et al., 2021). It relies on GPT-4o mini for generation and paraphrasing and on GPT-4o for automatic quality screening.
  • Best result: Seed 1.6 Vision is the top model at 40.65% overall accuracy, and every model struggles to pin down specific events and their participants.
  • Thinking vs. non-thinking models: thinking models, notably GPT-5 mini and Qwen3-VL, tend to give overly specific answers and handle unanswerable questions poorly. Non-thinking models are especially weak at counting and comparison.
  • Error analysis: the main failure types are wrong event recognition, imprecise fine-grained argument identification and comprehension, and event-centric hallucination. The paper's contribution list gives 4,127 questions rather than 4,217, so verify the exact count against the released data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ftadqtyt374

@inproceedings{wang2026meur,
  title = {{MEUR: A Benchmark for Evaluating Vision-Language Models on Multimodal Event Understanding and Reasoning}},
  author = {Wang, Zimu and Wang, Yuqi and Chen, Tong and Zeng, Changyu and Na, Hongbin and Han, Nijia and Xing, Fuyu and Chen, Qi and Wang, Qiufeng and Nguyen, Anh and Wang, Shuihua and Chen, Ling and Su, Jionglong and Zhang, Haiyang and Wang, Wei},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4ftadqtyt374},
  url = {https://doi.org/10.63317/4ftadqtyt374}
}

References

71 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025. Qwen3-vl technical report. find our card
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. find
  3. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, https://tlo.mit.edu/https://www.sigkg.cn/ccks2021/en/index.php/resource-track/ 6 https://github.com/zimuwangnlp/MEUR ↗
  4. Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. find our card
  5. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. 2025. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. find
  6. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. find
  7. Xiaoqiang Kang, Zimu Wang, Xiaobo Jin, Wei Wang, Kaizhu Huang, and Qiufeng Wang. 2025a. Template-driven llm-paraphrased framework for tabular math word problem generation. Proceedings of AAAI, 39(23):24303–24311. find
  8. Xiaoqiang Kang, Zimu Wang, Xiaochen Zi, Xiaobo Jin, Kaizhu Huang, Fei Yin, and Qiufeng Wang. 2026. A benchmark and method for photographed table reasoning. Pattern Recognition, 178:113355. find
  9. Xiaoqiang Kang, Shengen Wu, Zimu Wang, Yilin Liu, Xiaobo Jin, Kaizhu Huang, Wei Wang, Yutao Yue, Xiaowei Huang, and Qiufeng Wang. 2025b. Can grpo boost complex multimodal table understanding? find
  10. Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. 2025. Natural language understanding and inference with mllm in visual question answering: A survey. ACM Comput. Surv., 57(8). find
  11. Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In Proceedings of ACL, pages 2557–2568. find
  12. Ruochen Li, Zimu Wang, and Xinya Du. 2025a. Efficient document-level event relation extraction. In Proceedings of RepL4NLP, pages 92–99. find
  13. Ruosen Li, Ziming Luo, and Xinya Du. 2025b. Fg-prm: Fine-grained hallucination detection and mitigation in language model mathematical reasoning. find
  14. Li Lin, Yixin Cao, Lifu Huang, Shu’Ang Li, Xuming Hu, Lijie Wen, and Jianmin Wang. 2022. What makes the story forward? inferring commonsense explanations as prompts for future event generation. In Proceedings of SIGIR, page 1098–1109. find
  15. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. find
  16. Kang Liu, Yubo Chen, Jian Liu, Xinyu Zuo, and Jun Zhao. 2020. Extracting events and their relations from texts: A survey on recent research progress and challenges. AI Open, 1:22–39. find
  17. Yubo Ma, Zehao Wang, Mukai Li, Yixin Cao, Meiqi Chen, Xinze Li, Wenqi Sun, Kunquan Deng, Kun Wang, Aixin Sun, and Jing Shao. 2022. MMEKG: Multi-modal event knowledge graph towards universal representation across modalities. In Proceedings of ACL: System Demonstrations, pages 231–239. find
  18. Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. 2024. Fact-driven logical reasoning for machine reading comprehension. Proceedings of AAAI, 38(17):18851–18859. find
  19. Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou, and Juanzi Li. 2023a. When does in-context learning fall short and why? a study on specification-heavy tasks. find
  20. Hao Peng, Xiaozhi Wang, Feng Yao, Zimu Wang, Chuzhao Zhu, Kaisheng Zeng, Lei Hou, and Juanzi Li. 2023b. OmniEvent: A comprehensive, fair, and easy-to-use toolkit for event understanding. In Proceedings of EMNLP: System Demonstrations, pages 508–517. find
  21. Huan Rong, Zhongfeng Chen, Zhenyu Lu, Xiao ke Xu, Kai Huang, and Victor S. Sheng. 2025. Pred-id: Future event prediction based on event type schema mining by graph induction and deduction. Information Fusion, 117:102819. find
  22. Himanshu Sharma and Anand Singh Jalal. 2021. A survey of methods, datasets and evaluation metrics for visual question answering. Image and Vision Computing, 116:104327. find
  23. Zeno Vendler. 1957. Verbs and times. The Philosophical Review, 66(2):143–160. find
  24. Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024a. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. find
  25. Zimu Wang, Lei Xia, Wei Wang, and Xinya Du. 2024b. Document-level causal relation extraction with knowledge-guided binary question answering. In Findings of EMNLP, pages 16944–16955. find
  26. Xiang Wei, Yufeng Chen, Ning Cheng, Xingyu Cui, Jinan Xu, and Wenjuan Han. 2024. CollabKG: A learnable human-machine-cooperative information extraction toolkit for (event) knowledge graph construction. In Proceedings of LREC-COLING, pages 3490–3506. find
  27. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. find our card
  28. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of SIGIR, page 641–649. find our card
  29. Fuyu Xing, Zimu Wang, Wei Wang, and Haiyang Zhang. 2025. Benchmarking and improving lvlms on event extraction from multimedia documents. find
  30. Changyu Zeng, Yifan Wang, Zimu Wang, Wei Wang, Zhengni Yang, Muyi Bao, Jiming Xiao, Anh Nguyen, and Yutao Yue. 2025. Numina: A natural understanding benchmark for multidimensional intelligence and numerical reasoning abilities. find
  31. Jiazheng Zhu, Shaojuan Wu, Xiaowang Zhang, Yuexian Hou, and Zhiyong Feng. 2023. Causal intervention for mitigating name bias in machine reading comprehension. In Findings of ACL, pages 12837–12852. 7. find
  32. Language Resource References Asma Ben Abacha, Vivek Datla, Sadid A. Hasan, Dina Demner-Fushman, and Henning Müller. 2020. Overview of the vqa-med task at imageclef 2020: Visual question answering and generation in the medical domain. In Conference and Labs of the Evaluation Forum. find
  33. Tommaso Caselli and Piek Vossen. 2017. The event StoryLine corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of EventStory, pages 77–86. find
  34. Tong Chen, Zimu Wang, Yiyi Miao, Haoran Luo, Yuanfei Sun, Wei Wang, Zhengyong Jiang, Procheta Sen, and Jionglong Su. 2025. Medfact: A large-scale chinese dataset for evidence-based medical fact-checking of llm responses. find
  35. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of LREC, pages 4545–4552. find our card
  36. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of NAACL-HLT, pages 2368–2378. find our card
  37. Jesse Dunietz, Lori Levin, and Jaime Carbonell. 2017. The BECauSE corpus 2.0: Annotating causality and overlapping relations. In Proceedings of LAW, pages 95–104. find
  38. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2015. Overview of linguistic resources for the TAC KBP 2015 evaluations: Methodologies and results. In TAC. find
  39. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2016. Overview of Linguistic Resources for the TAC KBP 2016 Evaluations: Methodologies and Results. In TAC. find
  40. Joe Ellis, Jeremy Getman, and Stephanie M Strassel. 2014. Overview of linguistic resources for the TAC KBP 2014 evaluations: Planning, execution, and results. In TAC. find
  41. Jeremy Getman, Joe Ellis, Zhiyi Song, Jennifer Tracey, and Stephanie Strassel. 2017. Overview of linguistic resources for the tac kbp 2017 evaluations: Methodologies and results. In TAC. find
  42. Goran Glavaš, Jan Šnajder, Marie-Francine Moens, and Parisa Kordjamshidi. 2014. HiEve: A corpus for extracting event hierarchies from news stories. In Proceedings of LREC, pages 3678–3683. find
  43. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of COLING, pages 6609–6625. find our card
  44. Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of CVPR, pages 6693–6702. find our card
  45. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of CVPR, pages 1988–1997. find our card
  46. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10. find
  47. Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, and Juanzi Li. 2024a. MAVEN-FACT: A large-scale event factuality detection dataset. In Findings of EMNLP, pages 11140– 11158. find
  48. Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. 2018. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. In Proceedings of ECCV, pages 570–586. find
  49. Ruosen Li, Zimu Wang, Son Quoc Tran, Lei Xia, and Xinya Du. 2024b. Meqa: A benchmark for multi-hop event-centric question answering with explanations. In Advances in Neural Information Processing Systems, volume 37, pages 126835– 126862. find
  50. Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of NAACL-HLT, pages 894–908. find our card
  51. Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems, volume 27. find
  52. Paramita Mirza, Rachele Sprugnoli, Sara Tonelli, and Manuela Speranza. 2014. Annotating causality in the TempEval-3 corpus. In Proceedings of CAtoCL, pages 10–19. find
  53. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of CNS, pages 47–56. find our card
  54. Sameer S. Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. Ontonotes: A unified relational semantic representation. In Proceedings of ICSC 2007, pages 517–526. find
  55. Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded situation recognition. In Proceedings of ECCV, pages 314–332. find
  56. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of CVPR, pages 4631–4640. find
  57. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find our card
  58. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In Proceedings of SemEval, pages 1–9. find
  59. Marc Verhagen, Robert J. Gaizauskas, Frank Schilder, Mark Hepple, Jessica L. Moszkowicz, and James Pustejovsky. 2009. The tempeval challenge: identifying temporal relations in text. Language Resources and Evaluation, 43:161– 179. find
  60. Marc Verhagen, Roser Saurí, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 task 13: TempEval-2. In Proceedings of SemEval, pages 57–62. find
  61. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  62. ACE 2005 multilingual training corpus. Linguistic Data Consortium, 57. find
  63. Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2017. Explicit knowledge-based reasoning for visual question answering. In Proceedings of IJCAI-17, pages 1290–1296. find
  64. Shuo Wang, Qiushuo Zheng, Zherong Su, Chongn-ing Na, and Guilin Qi. 2021. Meed: A multimodal event extraction dataset. In Proceedings of CCKS, pages 288–294. find
  65. Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. 2022. MAVEN-ERE: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. In Proceedings of EMNLP, pages 926–941. find our card
  66. Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, Jie Zhou, and Juanzi Li. 2024. MAVEN-ARG: Completing the puzzle of all-in-one event understanding dataset with event argument annotation. In Proceedings of ACL, pages 4072–4091. find
  67. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of EMNLP, pages 1652–1671. find our card
  68. Zhenguo Yang, Jiale Xiang, Jiuxiang You, Qing Li, and Wenyin Liu. 2023. Event-oriented visual question answering: The e-vqa dataset and benchmark. IEEE Transactions on Knowledge and Data Engineering, 35(10):10210–10223. find
  69. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of EMNLP, pages 2369–2380. find our card
  70. Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, and Maosong Sun. 2022. LEVEN: A large-scale Chinese legal event detection dataset. In Findings of ACL, pages 183–201. find
  71. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of CVPR, pages 4995–5004. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-meur-benchmark-evaluating-vision-language/