Elamur Research
☰
home / papers / 2023 / MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2306.13394

Summary

Evaluations of multimodal large language models (MLLMs) have mostly relied on showcase examples, older captioning or VQA datasets that models may have seen in training, small open-ended sets, or probes of a single skill such as hallucination. MME is a broad benchmark meant to fill this gap. It spans 14 subtasks, split into perception (coarse- and fine-grained recognition plus OCR) and cognition (reasoning over what is seen combined with LLM knowledge).

All instruction–answer pairs were written by hand to reduce leakage. Public datasets supplied only images, and further images came from real photographs and image generation. Every image gets two yes/no questions with a fixed short prompt, so models can be compared without prompt engineering and scored automatically.

Across 30 MLLMs, results show substantial headroom. Reasoning subtasks are especially weak, and failures include ignoring instructions and hallucinating objects. The work is a NeurIPS 2025 Datasets & Benchmarks paper and has become a widely used reference point for comparing MLLMs.

Key points

  • Subtasks:
    • Perception has 10 subtasks: existence, count, position, color, poster, celebrity, scene, landmark, artwork and OCR.
    • Cognition has 4 subtasks, namely code reasoning, translating text, doing numerical calculations, and reasoning with commonsense.
  • Scoring:
    • Each subtask score is accuracy plus "accuracy+", where accuracy+ requires both questions about an image to be answered correctly. This gives a maximum of 200 per subtask.
    • Totals are 2000 for perception and 800 for cognition.
  • Leaders:
    • Perception: WeMM (1621.66), InfMLLM (1567.99) and SPHINX (1560.15).
    • Cognition: GPT-4V (517.14), Lion (445.71) and WeMM (445.00).
    • GPT-4V scores 170 in code reasoning. It gets zero on celebrity questions because it declines to identify people.
  • Reasoning ceiling: no model exceeds 150 in commonsense reasoning, numerical calculation or text translation.
  • Four recurring failures:
    • not following the yes/no instruction;
    • perception errors;
    • broken reasoning chains;
    • instruction-induced object hallucination, where always answering "yes" yields about 50% accuracy and near-zero accuracy+.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

386 papers of our library cite it (of the 140673 whose bibliographies we read); 31 of them have a page here, 355 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.13394 · arXiv 2306.13394

@misc{fu2023mme,
  title = {{MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models}},
  author = {Fu, Chaoyou and Chen, Peixian and Shen, Yunhang and Qin, Yulei and Zhang, Mengdan and Lin, Xu and Yang, Jinrui and Zheng, Xiawu and Li, Ke and Sun, Xing and Wu, Yunsheng and Ji, Rongrong and Shan, Caifeng and He, Ran},
  year = {2023},
  doi = {10.48550/arxiv.2306.13394},
  eprint = {2306.13394},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.13394}
}

References

66 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Infmllm. https://github.com/mightyzau/InfMLLM, 2023. ↗
  2. Lion. https://github.com/mynameischaos/Lion, 2023. ↗
  3. Octopus. https://github.com/gray311/UnifiedMultimodalInstructionTuning, 2023. ↗
  4. Skywork-mm. https://github.com/will-singularity/Skywork-MM, 2023. ↗
  5. Visualglm-6b. https://github.com/THUDM/VisualGLM-6B, 2023. ↗
  6. Wemm. https://github.com/scenarios/WeMM, 2023. ↗
  7. Xcomposer-vl. https://github.com/InternLM/InternLM-XComposer, 2023. ↗
  8. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022. find our card
  9. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint:2308.12966, 2023. find our card
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020. find
  11. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint:1504.00325, 2015. find our card
  12. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint:2305.06500, 2023. find our card
  13. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint:2303.03378, 2023. find
  14. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. NeurIPS, 2024. find
  15. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint:2304.15010, 2023. find
  16. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint:2305.04790, 2023. — [arXiv:2305.04790](https://arxiv.org/abs/2305.04790) find
  17. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. find our card
  18. Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint:2309.03905, 2023. — [arXiv:2309.03905](https://arxiv.org/abs/2309.03905) find
  19. Wenbo Hu, Yifan Xu, Y Li, W Li, Z Chen, and Z Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. arXiv preprint:2308.09936, 2023. — [arXiv:2308.09936](https://arxiv.org/abs/2308.09936) find
  20. Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In ECCV, 2020. find
  21. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint:2302.14045, 2023. find our card
  22. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint:2306.05425, 2023. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) find
  23. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint:2305.03726, 2023. find our card
  24. Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, and Yueting Zhuang. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. arXiv preprint:2308.04152, 2023. — [arXiv:2308.04152](https://arxiv.org/abs/2308.04152) find
  25. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint:2301.12597, 2023. find our card
  26. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint:2305.10355, 2023. find our card
  27. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. find our card
  28. Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint:2311.07575, 2023. — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) find
  29. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint:2306.14565, 2023. find our card
  30. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. preprint:2304.08485, 2023. find
  31. Jingying Liu, Binyuan Hui, Kun Li, Yunke Liu, Yu-Kun Lai, Yuxiang Zhang, Yebin Liu, and Jingyu Yang. Geometry-guided dense perspective network for speech-driven facial animation. IEEE TVCG, 2021. find
  32. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint:2307.06281, 2023. find our card
  33. Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detection via transverse and longitudinal sequence connection. PR, 2019. find
  34. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022. find
  35. Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv preprint:2305.15023, 2023. — [arXiv:2305.15023](https://arxiv.org/abs/2305.15023) find
  36. Hui Mao, Ming Cheung, and James She. Deepart: Learning joint representations of visual arts. In ICM, 2017. find
  37. Hui Mao, James She, and Ming Cheung. Visual arts search on mobile devices. TOMM, 2019. find
  38. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. find
  39. OpenAI. Gpt-4 technical report. arXiv preprint:2303.08774, 2023. find our card
  40. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint:2303.17580, 2023. find our card
  41. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint:2305.16355, 2023. find
  42. Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025. find
  43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint:2302.13971, 2023. find
  44. Xiaoguang Tu, Zhi He, Yi Huang, Zhi-Hao Zhang, Ming Yang, and Jian Zhao. An overview of large ai models and their applications. Visual Intelligence, 2024. find
  45. Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint:2205.14100, 2022. find
  46. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint:2305.11175, 2023. — [arXiv:2305.11175](https://arxiv.org/abs/2305.11175) find
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint:2201.11903, 2022. find our card
  48. Hao Wen, Hongbo Kang, Jian Ma, Jing Huang, Yuanwang Yang, Haozhe Lin, Yu-Kun Lai, and Kun Li. Dycrowd: Towards dynamic crowd reconstruction from a large-scene video. IEEE TPAMI, 2025. find
  49. Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In CVPR, 2020. find
  50. Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin. An early evaluation of gpt-4v (ision). arXiv preprint:2310.16534, 2023. Visual instruction tuning. arXiv — [arXiv:2310.16534](https://arxiv.org/abs/2310.16534) find
  51. Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint:2212.10773, 2022. find
  52. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint:2311.04257, 2023. find our card
  53. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint:2306.13549, 2023. find our card
  54. Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint:2310.16045, 2023. find
  55. Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024. — [arXiv:2404.16006](https://arxiv.org/abs/2404.16006) find
  56. Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint:2310.00653, 2023. — [arXiv:2310.00653](https://arxiv.org/abs/2310.00653) find
  57. Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training a gpt4-style language model with multimodal inputs? arXiv preprint:2307.02469, 2023. find
  58. Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. arXiv preprint:2305.01278, 2023. find
  59. Jinsong Zhang, Xiongzheng Li, Hailong Jia, Jin Li, Zhuo Su, Guidong Wang, and Kun Li. Logavatar: Local gaussian splatting for human avatar modeling from monocular video. CAD, 2025. find
  60. Jinsong Zhang, Minjie Zhu, Yuxiang Zhang, Zerong Zheng, Yebin Liu, and Kun Li. Speechact: Towards generating whole-body motion from speech. IEEE TVCG, 2025. find
  61. Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint:2309.07915, 2023. find
  62. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint:2303.18223, 2023. find our card
  63. Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. arXiv preprint:2305.16934, 2023. — [arXiv:2305.16934](https://arxiv.org/abs/2305.16934) find
  64. Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint:2305.16103, 2023. — [arXiv:2305.16103](https://arxiv.org/abs/2305.16103) find
  65. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. NeurIPS, 2014. find
  66. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint:2304.10592, 2023. find our card