Elamur Research⌕
☰
home / papers / 2025 / PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models

PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models

Paper · 2025 · Shi Qiu, Shaoyang Guo, Zhuo-Yang Song et al.

Summary

Popular reasoning benchmarks for LLMs are nearly saturated, often built from public material that models may have seen in training, and sometimes contain faulty items or scoring rules. PHYBench is a response to these problems. It is a set of 500 newly written physics problems, ranging from school exercises to Olympiad tasks, adapted and checked by 178 students from Peking University's School of Physics. Each problem describes a physical situation in text, and the model must produce a symbolic formula for a target quantity. To grade answers more finely than right or wrong, the authors propose the Expression Edit Distance (EED) Score. It compares SymPy expression trees of the predicted and reference answers using tree edit distance, which gives partial credit. Top models remain well below human performance. Error analysis shows that models often introduce wrong conditions or equations partway through a solution and rarely catch these mistakes themselves. The benchmark therefore offers a contamination-resistant, discriminative test of multi-step physical reasoning.

Key points

  • Gemini 2.5 Pro scores 36.9% accuracy, the best model result, while the human baseline of physics undergraduates reaches 61.9% accuracy (70.4 EED).
  • Compared with binary scoring, EED Score improves sample efficiency by 204%.
  • DeepSeek-R1 compared across benchmarks:
    • On PHYBench it averages about 10.6K output tokens and gets 25.0% right.
    • On MATH-500 it reaches 97.3%, and on AIME 2024 it reaches 79.8% with about 7.7K tokens.
  • Tests with deliberately corrupted reasoning traces reveal three behaviours: superficial, genuine, and pseudo-genuine reasoning.
    • The strongest systems, including DeepSeek-R1 and Gemini 2.5 Pro, withstand these errors mostly by relying on workarounds, such as dimensional or limiting-case checks and formal derivations.
    • They do not show true semantic understanding.
  • Stated limitations:
    • The problems are mostly Olympiad-level and unevenly spread across physics topics.
    • EED judges only the final answer, not the intermediate steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China335 works · 43,759 citations🎓Chinese University of Hong KongHong Kong · China176 works · 17,536 citations

Also on the byline: Beijing Computational Science Research Center.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Jiaming Ji, Weike Wang, Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing-Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia, Jiawei Lin, Zheyu Shen, Jiahang Chen, Qiuhao Xiong, Binran Wang, Fengyuan Wang, Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing-Hong Cao, Ming-xing Luo, Yaodong Yang, Muhan Zhang and Hua Xing Zhu. 2025. PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models. arXiv:2504.16074.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.16074 · arXiv 2504.16074

@misc{qiu2025phybench,
  title = {{PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models}},
  author = {Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Jiaming Ji, Weike Wang, Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing-Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia, Jiawei Lin, Zheyu Shen, Jiahang Chen, Qiuhao Xiong, Binran Wang, Fengyuan Wang, Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing-Hong Cao, Ming-xing Luo, Yaodong Yang, Muhan Zhang and Hua Xing Zhu},
  year = {2025},
  doi = {10.48550/arxiv.2504.16074},
  eprint = {2504.16074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.16074}
}

References

30 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL https://matharena.ai/. ↗
  2. David T. Barnard, Gwen Clarke, and Nicholas Duncan. Tree-to-tree correction for document trees: Technical report 95-372. Technical report, Dept. of Computing and Information Science, Queen’s University, Kingston, ON, Canada, 1995. find
  3. claude. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/claude-3 -7-sonnet, 2025. ↗
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  5. DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412.1 9437. ↗
  6. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. ↗
  7. P Team et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines, 2025. URL https://arxiv.org/abs/2502.14739. ↗
  8. Phan et al. Humanity’s Last Exam. working paper or preprint, January 2025. URL — Humanity's Last Exam https://hal.science/hal-04915593. ↗
  9. grok. Grok 3 beta — the age of reasoning agents. https://x.ai/news/grok-3, 2025. ↗
  10. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku,…
  11. Hugging Face H4. Aime 2024 dataset. https://huggingface.co/datasets/HuggingFaceH 4/aime_2024, 2024. Accessed: 2025-05-16. • Yizhe Zheng • Bowen Yu • Chengzhou Wu • Mengyao Zhang • Houcheng Li • Peilin Li • Yuyang Zhao • Bingru He • Zongyue Hou • Jiajun Yan • Lingrui Zhang • Jianyuan Luo ↗
  12. hynky1999. Latex2sympy_extended package. https://pypi.org/project/latex2sympy 2-extended/, 2018. Accessed: 2025-05-16. ↗
  13. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL — Let's Verify Step by Step https://openreview.net/forum?id=v8L0pN6EOi. ↗
  14. Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fab… — [doi:10.7717/peerj-cs.103](https://doi.org/10.7717/peerj-cs.103) find
  15. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
  16. OpenAI. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276. ↗
  17. OpenAI. Openai o1 system card, 2024. URL https://arxiv.org/abs/2412.16720. ↗
  18. OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/learnin g-to-reason-with-llms/. ↗
  19. OpenAI. Introducing gpt-4.1. https://openai.com/index/gpt-4-1/, 2025. ↗
  20. OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o 3-and-o4-mini/, 2025. ↗
  21. OpenAI. Openai o3-mini: Pushing the frontier of cost-effective reasoning. https://openai.com/index/openai-o3-mini/, 2025. ↗
  22. Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating llms on 2025 usa math olympiad, 2025. URL https://arxiv.org/abs/2503.21934. ↗
  23. Machel et al Reid. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 2024.
  24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98. ↗
  25. Gemini Team. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/2312.11805. ↗
  26. Qwen Team. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. ↗
  27. Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, 2025. URL https://qwenlm.github.io/blog/qwq-32b/. ↗
  28. Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/. ↗
  29. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL https://arxiv.org/abs/2203.11171. ↗
  30. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023. List of appendices find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qiu-2025-phybench-holistic-evaluation-physical-perception/