Elamur Research⌕
☰
home / papers / 2025 / PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning

PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning

Paper · 2025 · Xinyu Zhang, Yuxuan Dong, Yanrui Wu et al.

Summary

Existing physics benchmarks for large language models (e.g., ScienceQA, SciBench, GPQA) mostly use short solutions of about 3–4 formulas and score only the final answer. PhysReason fills this gap with 1,200 problems drawn from college entrance exams and physics olympiads. A quarter of the problems test knowledge, and the rest test reasoning at easy, medium and hard levels. Each problem is annotated with a diagram, step-by-step solution, step analysis, the theorems used and a difficulty label.

To score answers, the authors built the Physics Solution Auto Scoring Framework (PSAS). It has a fast answer-level mode (PSAS-A) and a detailed step-level mode (PSAS-S), and its judgements agree with ground truth more than 98% of the time.

Fifteen models were tested, and even the strongest reasoning-oriented ones score below 60%. Accuracy falls steadily as solutions get longer. Step-level analysis shows that models break down mainly on:

  • applying theorems,
  • understanding physical processes,
  • calculation,
  • analysing conditions.

This makes the benchmark useful for diagnosing where reasoning fails, not only whether it does.

Key points

  • Data and coverage: 1,200 problems (300 knowledge plus 300 each of easy, medium and hard reasoning), covering 147 physics theorems. About 81% of problems include diagrams. Solutions average 8.1 steps, rising to 15.6 for hard problems.
  • Sources and filtering: Problems came from 1,254 PDFs holding over 20,000 problems, including Chinese, Indian and Russian exams, IPhO, APhO and EPhO. PDFs were parsed with MinerU, and translations were checked by experts. Problems solvable by a five-minute Google search were removed.
  • Top models: Deepseek-R1, Gemini-2.0-Flash-Thinking and o3-mini-high lead but stay under 60% at answer level. Accuracy drops from 75.11% on knowledge problems to 31.95% on hard ones.
  • Error feedback: Pointing a model to its first error with PSAS-S raised accuracy by 3–6%. Simply re-feeding its previous reasoning lowered accuracy by 3–5%.
  • Release: Data and code are released under MIT and CC BY-NC-SA licenses; the main stated limitation is that the problems assume idealized physical conditions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang and Jun Liu. 2025. PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.811

@inproceedings{zhang2025physreason,
  title = {{PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning}},
  author = {Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang and Jun Liu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.811},
  url = {https://doi.org/10.18653/v1/2025.acl-long.811}
}

References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Claude 3.5 sonnet. Daman Arora, Himanshu Gaurav Singh, et al. Have llms advanced enough? a challenging problem solving benchmark for large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. find
  2. Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444. — [arXiv:2501.05444](https://arxiv.org/abs/2501.05444) find
  3. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889–7901.
  4. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proce…
  5. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
  6. Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. 2025. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501.16411. — [arXiv:2501.16411](https://arxiv.org/abs/2501.16411) find
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  8. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/ opencompass. ↗
  9. Google Deepmind. Gemini 2.0 flash thinking mode. find
  10. Google Deepmind. 2024. Introducing gemini 2.0: Our new ai model for the agentic era. find
  11. Google Deepmind. 2025. Introducing gemini 2.0 pro. find
  12. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  13. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. 2024a. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985.
  14. Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2024b. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462–12469. IEEE. find
  15. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  17. Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, and Min Yang. 2024. E-EVAL: A comprehensive Chinese k-12 education evaluation benchmark for large language models. In Findings of the Association for Computational Linguisti…
  18. Yu Huang, Yue Chen, and Zhu Li. 2023. Applications of large scale foundation models for autonomous driving. arXiv preprint arXiv:2311.12144. — [arXiv:2311.12144](https://arxiv.org/abs/2311.12144) find
  19. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36.
  20. Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42.
  21. Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James Kwok. 2024. Forward-backward reasoning in large language models for mathematical verification. In Findings of the Association for Computational Linguistics ACL 2024, pages 6647–6661. find
  22. Morris Kline. 1981. Mathematics and the physical world. Courier Corporation. find
  23. Wenqiang Lai, Tianwei Zhang, Tin Lun Lam, and Yuan Gao. 2024. Vision-language model-based physical reasoning for robot liquid perception. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9652–9659. IEEE. find
  24. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469.
  25. Qwen Team. 2024b. Qwq: Reflect deeply on the boundaries of the unknown. find
  26. Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, et al. 2024. Controllable text generation for large language models: A survey. arXiv preprint arXiv:2408.12599.
  27. Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. 2024a. Mineru: An open-source solution for precise document content extraction. Preprint, arXiv:240… find
  28. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  29. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. find
  30. OpenAI. 2024a. Learning to reason with LLMs. find
  31. OpenAI. 2024b. Openai o1-mini. find
  32. OpenAI. 2025. Openai o3-mini. find
  33. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  34. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling llm test-time compute optimally can be more effective than scaling model parameters. In International Conference on Learning Representations.
  35. Seonil Son, Ju-Min Oh, Heegon Jin, Cheolhun Jang, Jeongbeom Jeong, and Kuntae Kim. 2024. Varco arena: A tournament approach to reference-free benchmarking large language models. arXiv preprint arXiv:2411.01281. — [arXiv:2411.01281](https://arxiv.org/abs/2411.01281) find
  36. Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, and Rui Yan. 2024a. Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa… find
  37. Zaharov Timur, Konstantin Korolev, and Aleksandr Nikolich. 2024. Physics big. find
  38. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  39. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. In Forty-first International Conference on Machine Learning.
  40. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2025. Are large language models really good logical reasoners? a comprehensive evaluation and beyond. IEEE Transactions on Knowledge and Data Engineering.
  41. Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, Shuai Yuan, Qika Lin, Yu Qiao, and Jun Liu. 2024. Symbol-LLM: Towards foundational symbol-centric interface for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… find
  42. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
  43. Lingling Zhang, Yifei Li, Qianying Wang, Yun Wang, Hang Yan, Jiaxin Wang, and Jun Liu. 2024. Fprompt-plm: Flexible-prompt on pretrained language model for continual few-shot relation extraction. IEEE Transactions on Knowledge and Data Engineering, 36(12):8267–8282. find
  44. Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2024b. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061. find
  45. Lingling Zhang, Yujie Zhong, Qinghua Zheng, Jun Liu, Qianying Wang, Jiaxin Wang, and Xiaojun Chang. 2025. Tdgi: Translation-guided double-graph inference for document-level relation extraction. IEEE Transactions on Pattern Analysis & Machine Intelligence. find
  46. Qwen Team. 2024a. Qvq: To see the world with wisdom. find
  47. Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. 2024. Docmath-eval: Evaluating math reasoning capabilities of llms in understanding long and specialized documents. In Proceedings of the 62nd Annual Meeting of the Associat… find
  48. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559.
  49. ZhipuAI. 2024. Glm-zero mode. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-physreason-comprehensive-benchmark-physics-based/