Elamur Research⌕
☰
home / papers / 2025 / HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?

HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?

Paper · 2025 · Fangchen Yu, Haiyuan Wan, Qianjia Cheng et al.

Summary

Existing physics benchmarks for (M)LLMs do not track recent Olympiad exams, often omit diagrams, and report plain accuracy. As a result, they cannot show how models compare with actual student competitors. HiPhO fills this gap. It assembles 13 full high school physics Olympiad papers from 2024–2025, covering international and regional contests and problems ranging from text-only to diagram-heavy. Each problem is graded twice, once on the final answer and once on intermediate steps using the official marking schemes, and the higher of the two counts. Total exam scores are then placed against medal cutoffs derived from human results, so a model effectively earns gold, silver or bronze. Thirty current models were tested. Closed-source reasoning MLLMs reach gold on 6 to 12 exams. Some open-source reasoning models also collect several golds, but open-source chat MLLMs generally stay at bronze or lower. Even the strongest systems still score below the best students on hard contests like IPhO and EuPhO. The benchmark therefore offers a realistic, human-referenced measure of multimodal physical reasoning.

Key points

  • Dataset scale: 360 problems and 519 subquestions from 13 Olympiads, including IPhO, APhO, EuPhO, NBPhO, PanPhO, PanMechanics and F=MA. Each problem is labeled with one of five physics domains, one of four modality categories, and one of six answer formats.
  • Medal cutoffs: These come from medalists' theory-exam scores. For F=MA, which has no official medals, the USAPhO qualifying cutoff serves as gold, and silver and bronze follow Physics Bowl percentile conventions.
  • Grading method: Step-level grading is model-based. On IPhO 2024, Gemini-2.5-Flash as grader tracked human experts closely (24.6 vs. 24.8 for Gemini-2.5-Pro's solutions), while GPT-4o was much less accurate.
  • Evaluation setup: 30 models were tested (11 closed-source MLLMs, 11 open-source MLLMs, 8 open-source LLMs) through VLMEvalKit, with 8 runs per problem at temperature 0.6.
  • IPhO 2025 result: The top human scored 29.2 in theory, against 22.7 for the best model (gold line 19.7). Intern-S1 and DeepSeek-R1 earned 4–8 golds each across the benchmark. Code is at github.com/SciYu/HiPhO, with a leaderboard at phyarena.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksSocial Sciences · 2,009 worksEducation · 221 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Fangchen Yu, Haiyuan Wan, Qianjia Cheng, Yuchen Zhang, Jiacheng Chen, Fujun Han, Yulun Wu, Junchi Yao, Ruilizhen Hu, Ning Ding, Yu Cheng, Tao Chen, Lei Bai, Dongzhan Zhou, Yun Luo, Ganqu Cui and Peng Ye. 2025. HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?. arXiv:2509.07894.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.07894 · arXiv 2509.07894

@misc{yu2025hipho,
  title = {{HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?}},
  author = {Fangchen Yu, Haiyuan Wan, Qianjia Cheng, Yuchen Zhang, Jiacheng Chen, Fujun Han, Yulun Wu, Junchi Yao, Ruilizhen Hu, Ning Ding, Yu Cheng, Tao Chen, Lei Bai, Dongzhan Zhou, Yun Luo, Ganqu Cui and Peng Ye},
  year = {2025},
  doi = {10.48550/arxiv.2509.07894},
  eprint = {2509.07894},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.07894}
}

References

27 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025.
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025.
  3. Anthropic. Claude 3.7 sonnet system card. URL — Intern-S1: A Scientific Multimodal Foundation Model https://www.anthropic.com/news/ visible-extended-thinking. Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, Yu Cheng, Yu Cheng, Pei Chu, Tao Chu, Erfei Cui, Ganqu Cui, Long Cui… ↗
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint a…
  6. Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198–11201, 2024… find
  7. Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. Physics: Benchmarking foundation models on university-level physics problem solving. arXiv preprint arXiv:2503.21821, 2025. find
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  9. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Assoc…
  10. Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37: 19209–19253, 2024.
  11. Hynek Kydlı́ček. Math-Verify: Math Verification Library. huggingface/math-verify. URL https://github.com/Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,… ↗
  12. Meta. Llama 4 system card. URL — Humanity's Last Exam https://www.llama.com/docs/ model-cards-and-prompt-formats/llama4/. Mistral. Mistral-medium-3 system card. mistral-medium-3. URL https://mistral.ai/news/OpenAI. Gpt-4o system gpt-4o-system-card/. card, a. URL https://openai.com/index/OpenAI. Gpt-5 system gpt-5-system-c… ↗
  13. Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, et al. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074, 2025.
  14. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
  15. Hui Shen, Taiqiang Wu, Qi Han, Yunta Hsieh, Jizhou Wang, Yuyue Zhang, Yuxin Cheng, Zijian Hao, Yuansheng Ni, Xin Wang, et al. Phyx: Does your model have the “wits” for physical reasoning? arXiv preprint arXiv:2505.15929, 2025. — [arXiv:2505.15929](https://arxiv.org/abs/2505.15929) find
  16. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025a.
  17. V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan… — GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
  18. Lintao Wang, Encheng Su, Jiaqi Liu, Pengze Li, Peng Xia, Jiabei Xiao, Wenlong Zhang, Xinnan Dai, Xi Chen, Yuan Meng, et al. Physunibench: An undergraduate-level physics reasoning benchmark for multimodal models. arXiv preprint arXiv:2506.17667, 2025a. — [arXiv:2506.17667](https://arxiv.org/abs/2506.17667) find
  19. Weida Wang, Dongchen Huang, Jiatong Li, Tengchao Yang, Ziyang Zheng, Di Zhang, Dong Han, Benteng Chen, Binzhao Luo, Zhiyu Liu, et al. Cmphysbench: A benchmark for evaluating large language models in condensed matter physics. arXiv preprint arXiv:2508.18124, 2025b. — [arXiv:2508.18124](https://arxiv.org/abs/2508.18124) find
  20. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. xAI. Grok 4 system card. URL https:/…
  21. Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, et al. Seephys: Does seeing help thinking?–benchmarking vision-based physics reasoning. arXiv preprint arXiv:2505.19099, 2025. find
  22. Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin ZHANG, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. In Forty-second International Conference on Machine Learning, 2025.
  23. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  24. Yixin Ye, Yang Xiao, Tiantian Mi, and Pengfei Liu. Aime-preview: A rigorous and immediate evaluation framework for advanced mathematical reasoning. https://github.com/ GAIR-NLP/AIME-Preview, 2025. GitHub repository. ↗
  25. Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning. ACL, 2025.
  26. Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu, Lei Bai, Dongzhan Zhou, Ganqu Cui, and Peng Ye. Scaling physical reasoning with the physics dataset. arXiv preprint arXiv:2506.00022, 2025. — [arXiv:2506.00022](https://arxiv.org/abs/2506.00022) find
  27. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2025-hipho-far-m-llms-humans/