Elamur Research⌕
☰
home / papers / 2024 / GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation

Paper · 2024 · Yi Zong and Xipeng Qiu

Summary

Most vision-language benchmarks test basic perception or commonsense with short prompts. Chinese versions are usually translations of English datasets. As a result, they reveal little about how large vision-language models (LVLMs) cope with demanding, natively Chinese material. Zong and Qiu fill this gap with GAOKAO-MM, built from image-based multiple-choice items from China's college entrance exam (GAOKAO) papers from 2010 to 2023.

The authors extracted items manually with OCR support and rewrote formulas in LaTeX. They kept only items that cannot be solved without the image, removed duplicates, and upscaled small images with Lanczos resampling. Every item includes a human-written explanation.

Ten LVLMs were tested zero-shot, using subject-specific prompts and rule-based answer extraction. None reached 50% accuracy, and all open-source systems stayed below 25%. Mathematics, physics and the long Chinese-language items proved hardest. Scores also shifted noticeably between exam years, which suggests weak robustness.

The resource offers a tough, culturally grounded yardstick for multilingual multimodal models. It may also inform educational applications.

Key points

  • GAOKAO-MM contains 646 questions across 8 subjects: Chinese, math, physics, chemistry, biology, politics, history and geography. It includes 897 images of 12 types, such as maps, diagrams, function graphs and geometric shapes.
  • The average question length is 260.19, more than four times that of MMMU (59.33). The text is native Chinese rather than translated, unlike MM-Bench-CN.
  • The top results were GPT-4V at 48.1%, Qwen-VL-Plus at 41.2% and Gemini-Pro-Vision at 35.1%. The best open-source model was mPLUG-Owl2 at 24.1%, and the weakest was CogVLM-Chat at 13.2%.
  • The top models did better than their own averages on maps but struggled with function graphs and geometry. GPT-4V stood out on flow charts. A single model's accuracy could vary by up to 20% across exam years.
  • The data and evaluation code are released at github.com/OpenMOSS/GAOKAO-MM, and the authors plan yearly updates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Yi Zong and Xipeng Qiu. 2024. GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation. arXiv:2402.15745.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.15745 · arXiv 2402.15745

@misc{zong2024gaokao,
  title = {{GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation}},
  author = {Yi Zong and Xipeng Qiu},
  year = {2024},
  doi = {10.48550/arxiv.2402.15745},
  eprint = {2402.15745},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.15745}
}

References

24 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, and Dmitry Lepikhin et al. 2023. Palm 2 technical report.
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4.
  5. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling.
  6. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models.
  7. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering.
  8. Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Large multilingual models pivot zero-shot multimodal learning across languages.
  9. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. 2018. Figureqa: An annotated figure dataset for visual reasoning. find
  10. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images.
  11. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning.
  12. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023b. Mmbench: Is your multi-modal model an all-around player?
  13. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
  14. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022a. Learn to explain: Multimodal reasoning via thought chains for science question answering. find
  15. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2022b. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. find
  16. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. find
  17. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774.
  18. ArXiv, Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. find
  19. Gemini Team, Rohan Anil, and Sebastian Borgeaud et al. 2023. Gemini: A family of highly capable multimodal models.
  20. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. Cogvlm: Visual expert for pretrained language models.
  21. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.
  22. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities.
  23. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mmmu: A massive multi-disc…
  24. Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. Evaluating the performance of large language models on gaokao benchmark.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zong-2024-gaokao-mm-chinese-human/