Elamur Research⌕
☰
home / papers / 2026 / Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark

Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark

Paper · 2026 · Kai Zou, Ziqi Huang, Yuhao Dong et al.

Summary

Unified multimodal models are meant to both interpret and produce images. Existing benchmarks, however, mostly test these two skills separately, or only check loose links between them such as content association or consistency. Uni-MMMU fills this gap with eight reasoning-heavy tasks where one ability logically depends on the other.

The tasks fall into two directions:

  • Generation aids understanding: the model draws intermediate visual states and then reasons over them (mazes, sliding puzzles, geometry with auxiliary lines, jigsaw completion).
  • Understanding guides generation: the model must know something first and then draw it (science state changes in physics, chemistry and biology; rendering images from SVG code).

Every item has a deterministic solution path and a single correct answer. This allows automatic scoring of both intermediate steps and final results, using programmatic parsers, perceptual metrics and an LLM-as-a-Judge.

Tests on unified, generation-only and understanding-only models show three things. The synergy pays off most when the dependency between steps is strict. Even imperfect self-generated intermediates beat decoupled pipelines. Image generation is the main weakness of current unified models.

Key points

  • Data construction:
    • Mazes are procedurally generated 6×6 perfect mazes, built with DFS carving and checked with BFS. Only mazes with a unique shortest path of length 2–10 are kept.
    • Sliding puzzles are 3×3 8-puzzles in a fixed 9-color palette, each with a unique optimal solution.
    • Geometry problems are taken from the Geo-Laux benchmark.
    • Science items come from an LLM-driven pipeline followed by manual curation.
  • Oracle intermediates: supplying ground-truth intermediate states gives consistent, sizeable gains on all tasks. The biggest improvements appear in how accurately Maze problems are solved and how correct the generated Science images are.
  • Code Rendering: the GPT-4.1 → GPT-image pipeline shows a direct causal handoff. A correct SVG analysis yields a faithful image, while analysis errors carry straight into visual errors.
  • Typical failures:
    • Bagel inserts junk glyphs into maze states.
    • OmniGen2 copies the jigsaw reference image instead of completing it.
    • Ovis-U1 misreads SVG attributes.
    • Qwen-Image-Edit draws text-only fields onto the image.
  • Release: code, data, evaluation tools and judge configurations are available at github.com/uni-mmmu/Uni-MMMU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Kai Zou, Ziqi Huang, Yuhao Dong, Shulin Tian, Dian Zheng, Hongbo Liu, Jingwen He, Bin Liu, Yu Qiao and Ziwei Liu. 2026. Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.40

@inproceedings{zou2026uni,
  title = {{Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark}},
  author = {Kai Zou, Ziqi Huang, Yuhao Dong, Shulin Tian, Dian Zheng, Hongbo Liu, Jingwen He, Bin Liu, Yu Qiao and Ziwei Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.40},
  url = {https://doi.org/10.18653/v1/2026.acl-long.40}
}

References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. AIDC-AI. Ovis-u1: Unified understanding, generation, and editing. https://github.com/ AIDC-AI/Ovis-U1. Accessed 2025. ↗
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, and 1 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923.
  4. Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, and 1 others. 2025. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742. — [arXiv:2506.15742](https://arxiv.org/abs/2506.15742) find
  5. Dongping Chen, Ruoxi Chen, Shu Pu, Zhaoyi Liu, Yanru Wu, Caixi Chen, Benlin Liu, Yue Huang, Yao Wan, Pan Zhou, and 1 others. 2024. Interleaved scene graphs for interleaved text-and-image generation assessment. arXiv preprint arXiv:2411.17188. find
  6. VBench++: Comprehensive and versatile benchmark suite for video generative models. arXiv preprint arXiv:2411.13503. — [arXiv:2411.13503](https://arxiv.org/abs/2411.13503) find
  7. Junying Chen, Zhenyang Cai, Pengcheng Chen, Shunian Chen, Ke Ji, Xidong Wang, Yunjin Yang, and Benyou Wang. 2025a. Sharegpt-4o-image: Aligning multimodal models with gpt-4o-level image generation. arXiv preprint arXiv:2506.18095. find
  8. Adam Lerer, Adam P. Goucher, Aditya Perelman, Aidan Ramesh, Aidan Clark, A. J. Ostrow, Akila Welihinda, Alec Radford, Aleksander M adry, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. System Card, OpenAI.
  9. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025b. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811.
  10. Han Li, Xinyu Peng, Yaoming Wang, Zelin Peng, Xin Chen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Wenrui Dai, and Hongkai Xiong. 2025a. Onecat: Decoder-only auto-regressive model for unified understanding and generation. Preprint, arXiv:2509.03498. — [arXiv:2509.03498](https://arxiv.org/abs/2509.03498) find
  11. Gemini Team Google DeepMind. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical report, Google DeepMind. Technical report describing Gemini 2.5 Pro.
  12. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. 2025a. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. — [arXiv:2505.14683](https://arxiv.org/abs/2505.14683) find
  13. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. 2025b. Emerging properties in unified multimodal pretraining. Preprint, arXiv:2505.14683. — [arXiv:2505.14683](https://arxiv.org/abs/2505.14683) find
  14. Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and machines, 30(4):681–694. find
  15. Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. 2023. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344. — [arXiv:2306.09344](https://arxiv.org/abs/2306.09344) find
  16. Yumeng Fu, Jiayin Zhu, Lingling Zhang, Bo Zhao, Shaoxuan Ma, Yushun Zhang, Yanrui Wu, and Wenjun Wu. 2025. Geolaux: A benchmark for evaluating mllms’ geometry performance on long-step problems requiring auxiliary lines. arXiv preprint arXiv:2508.06226. find
  17. Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024a. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of… find
  18. Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024b. find
  19. Yi Li, Haonan Wang, Qixiang Zhang, Boyu Xiao, Chenchang Hu, Hualiang Wang, and Xiaomeng Li. 2025b. Unieval: Unified holistic evaluation for unified multimodal understanding and generation. arXiv preprint arXiv:2505.10483. — [arXiv:2505.10483](https://arxiv.org/abs/2505.10483) find
  20. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916.
  21. Yuanhuiyi Lyu, Chi Kit Wong, Chenfei Liao, Lutao Jiang, Xu Zheng, Zexin Lu, Linfeng Zhang, and Xuming Hu. 2025. Understanding-in-generation: Reinforcing generative capability of unified model via infusing understanding into generation. Preprint, arXiv:2509.18639. — [arXiv:2509.18639](https://arxiv.org/abs/2509.18639) find
  22. Sabbir Mollah, Rohit Gupta, Sirnam Swetha, Qingyang Liu, Ahnaf Munir, and Mubarak Shah. 2025. The telephone game: Evaluating semantic drift in unified models. arXiv preprint arXiv:2509.04438. — [arXiv:2509.04438](https://arxiv.org/abs/2509.04438) find
  23. Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kunpeng Ning, Bin Zhu, and 1 others. 2025. Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265. — [arXiv:2503.07265](https://arxiv.org/abs/2503.07265) find
  24. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205.
  25. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. find
  26. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Sam Ghasemipour, Alexander M. Rush, and 1 others. 2022. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487.
  27. Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, and Yi R. Fung. 2025. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers. Preprint, arXiv:2506.23918. — [arXiv:2506.23918](https://arxiv.org/abs/2506.23918) find
  28. Qwen Team and collaborators. 2025. Qwen-image: Crafting with native text rendering. arXiv preprint arXiv:2508.02324. Technical report / preprint.
  29. Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, and 1 others. 2024. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869. — [arXiv:2409.18869](https://arxiv.org/abs/2409.18869) find
  30. Hongyang Wei, Baixin Xu, Hongbo Liu, Cyrus Wu, Jie Liu, Yi Peng, Peiyu Wang, Zexiang Liu, Jingwen He, Yidan Xietian, Chuanxin Tang, Zidong Wang, Yichen Wei, Liang Hu, Boyi Jiang, William Li, Ying He, Yang Liu, Xuchen Song, and 2 others. 2025. Skywork unipic 2.0: Building kontext model with online rl… — [arXiv:2509.04548](https://arxiv.org/abs/2509.04548) find
  31. C. Wu and 1 others. 2025. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506.18871. expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567. — [arXiv:2506.18871](https://arxiv.org/abs/2506.18871) find
  32. Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, and Ziwei Liu. 2024. Evaluation agent: Efficient and promptable evaluation framework for visual generative models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025. find
  33. Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Xiaorong Zhu, Hao Li, Wenhao Chai, Zicheng Zhang, Renqiu Xia, Guangtao Zhai, Junchi Yan, and 1 others. 2025. Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing. arXiv preprint arXiv:2504.02826. — [arXiv:2504.02826](https://arxiv.org/abs/2504.02826) find
  34. Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. 2025. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755. — [arXiv:2503.21755](https://arxiv.org/abs/2503.21755) find
  35. Pengfei Zhou, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang, Ziyao Guo, Hao Zhang, Yuqi Lin, Yefei He, and 1 others. 2025. Opening: A comprehensive benchmark for judging open-ended interleaved image-text generation. In Proceedings of the Computer Vision and Pattern Recognition Confer… find
  36. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, and 1 others. 2024. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429. — [arXiv:2409.04429](https://arxiv.org/abs/2409.04429) find
  37. Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. 2025a. Show-o2: Improved native unified multimodal models. Preprint, arXiv:2506.15564. — [arXiv:2506.15564](https://arxiv.org/abs/2506.15564) find
  38. Wulin Xie, Yi-Fan Zhang, Chaoyou Fu, Yang Shi, Bingyan Nie, Hongkai Chen, Zhang Zhang, Liang Wang, and Tieniu Tan. 2025b. Mme-unify: A comprehensive benchmark for unified multimodal understanding and generation models. arXiv preprint arXiv:2504.03641. — [arXiv:2504.03641](https://arxiv.org/abs/2504.03641) find
  39. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  40. Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. 2025. Imgedit: A unified image editing dataset and benchmark. Preprint, arXiv:2505.20275. — [arXiv:2505.20275](https://arxiv.org/abs/2505.20275) find
  41. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for *Figure 4: Details of Maze Puzzle.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2026-uni-mmmu-massive-multi-discipline/