Elamur Research⌕
☰
home / papers / 2024 / Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents

Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents

Paper · 2024 · Raj Jaiswal, Dhruv Jain, Harsh Parimal Popat et al.

Summary

Open-source LLMs often fail at multi-step physics problems. Their chain-of-thought solutions typically go wrong in three ways: they misread the question or its given values, pick an unsuitable concept or formula, or make arithmetic and algebra slips. Several of these errors often appear in the same solution. The authors propose MoRA (Mixture of Refinement Agents), which avoids costly annotation and fine-tuning. In each iteration, GPT-4o inspects a solution, raises flags and assigns scores for each error type. Prioritized routing then calls specialized agents to fix the detected problems:

  • instruction prompting with error feedback for misreadings,
  • retrieval-style "thought" generation for concept errors,
  • code generation and execution for calculations.

Refinement repeats until no errors remain. For evaluation, the authors built PhysicsQA, a set of intermediate-level high school physics problems with step-by-step solutions. They also tested on SciEval and MMLU physics subsets. MoRA substantially improves Llama-3-70B and Gemma-2-27B. The work shows that a strong model used only as an error detector can narrow the gap between open models and GPT-4o.

Key points

  • PhysicsQA has 370 questions across six topics. The largest is Electromagnetism (29.8%), followed by Mechanics and Kinematics (21.8%).
  • On PhysicsQA, MoRA improves on CoT baselines by 13.38% for Llama-3-70B and 16.03% for Gemma-2-27B. Across benchmarks, gains reach up to about 16%.
  • Evaluation covers four datasets: PhysicsQA, SciEval-Static, MMLU High School and MMLU College.
  • In error analysis on PhysicsQA, the two open models average 21.62% of answers with computational mistakes and 18.11% with conceptual errors. GPT-4o makes far fewer errors.
  • Ablations by refinement rate show code-based refinement works well, fixing about 73% of computational errors on PhysicsQA for both models. Concept errors are harder, with roughly 47–49% fixed. The table labels in the extracted text appear scrambled, so these per-agent figures follow the paper's prose.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Raj Jaiswal, Dhruv Jain, Harsh Parimal Popat, Avinash Anand, Abhishek Dharmadhikari, Atharva Marathe and Rajiv Ratn Shah. 2024. Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents. arXiv:2412.00821.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.00821 · arXiv 2412.00821

@misc{jaiswal2024improving,
  title = {{Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents}},
  author = {Raj Jaiswal, Dhruv Jain, Harsh Parimal Popat, Avinash Anand, Abhishek Dharmadhikari, Atharva Marathe and Rajiv Ratn Shah},
  year = {2024},
  doi = {10.48550/arxiv.2412.00821},
  eprint = {2412.00821},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.00821}
}

References

41 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anand, A.; Addala, K.; Baghel, K.; Goel, A.; Hira, M.; Gupta, R.; and Shah, R. R. 2023a. Revolutionizing High School Physics Education: A Novel Dataset. In Goyal, V.; Kumar, N.; Bhowmick, S. S.; Goyal, P.; Goyal, N.; and Kumar, D., eds., Big Data and Artificial Intelligence, 64– 79. Cham: Springer N… find
  2. Anand, A.; Goel, A.; Hira, M.; Buldeo, S.; Kumar, J.; Verma, A.; Gupta, R.; and Shah, R. R. 2023b. Sciphyrag-retrieval augmentation to improve llms on physics q &a. In International Conference on Big Data Analytics, 50–63. Springer. find
  3. Anand, A.; Gupta, M.; Prasad, K.; Goel, U.; Lal, N.; Verma, A.; and Shah, R. R. 2023c. KG-CTG: citation generation through knowledge graph-guided large language models. In International Conference on Big Data Analytics, 37–49. Springer. find
  4. Anand, A.; Gupta, M.; Prasad, K.; Singla, N.; Sanjeev, S.; Kumar, J.; Shivam, A. R.; and Shah, R. R. 2024a. Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks. arXiv preprint arXiv:2404.13099. find
  5. Anand, A.; Jairath, A.; Lal, N.; Bangar, S.; Sikka, J.; Verma, A.; Shah, R. R.; and Satoh, S. 2023d. GEC-DCL: Grammatical Error Correction Model with Dynamic Context Learning for Paragraphs and Scholarly Papers. In International Conference on Big Data Analytics, 95–110. Springer. find
  6. Anand, A.; Jaiswal, R.; Dharmadhikari, A.; Marathe, A.; Popat, H.; Mital, H.; Nair, A. R.; Prasad, K.; Kumar, S.; Verma, A.; et al. 2024b. GeoVQA: A Comprehensive Multimodal Geometry Dataset for Secondary Education. In 2024 IEEE 7th International Conference on Multimedia Information Processing and R… find
  7. Anand, A.; Kapuriya, J.; Kirtani, C.; Singh, A.; Saraf, J.; Lal, N.; Kumar, J.; Shivam, A. R.; Verma, A.; Shah, R. R.; et al. 2024c. MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering. arXiv preprint arXiv:2404.12926. — [arXiv:2404.12926](https://arxiv.org/abs/2404.12926) find
  8. Anand, A.; Kapuriya, J.; Singh, A.; Saraf, J.; Lal, N.; Verma, A.; Gupta, R.; and Shah, R. 2024d. MM-PhyQA: Multimodal Physics Question-Answering with Multi-image CoT Prompting. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, 53–64. Springer. find
  9. Anand, A.; Prasad, K.; Goel, U.; Gupta, M.; Lal, N.; Verma, A.; and Shah, R. R. 2023e. Context-enhanced language models for generating multi-paper citations. In International Conference on Big Data Analytics, 80–94. Springer. find
  10. Arora, D.; Singh, H. G.; et al. 2023. Have llms advanced enough? a challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074.
  11. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877– 1901.
  12. Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. find
  13. Chern, E.; Zou, H.; Li, X.; Hu, J.; Feng, K.; Li, J.; and Liu, P. 2023. Generative ai for math: Abel. URL https://github.com/GAIR-NLP/abel. ↗
  14. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  15. Edge, D.; Trinh, H.; Cheng, N.; Bradley, J.; Chao, A.; Mody, A.; Truitt, S.; and Larson, J. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. find
  16. Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations.
  17. Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, 10764–10799. find
  18. PMLR. He, C.; Luo, R.; Bai, Y.; Hu, S.; Thai, Z. L.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
  19. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.;…
  20. Li, X.; Wang, W.; Li, M.; Guo, J.; Zhang, Y.; and Feng, F. 2024. Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction. arXiv preprint arXiv:2406.00755. find
  21. Ling, Z.; Fang, Y.; Li, X.; Huang, Z.; Lee, M.; Memisevic, R.; and Su, H. 2024. Deductive verification of chain-of-thought reasoning. Advances in Neural Information Processing Systems, 36.
  22. Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2521. find
  23. Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; and Zhang, D. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
  24. Ma, Y.; Gou, Z.; Hao, J.; Xu, R.; Wang, S.; Pan, L.; Yang, Y.; Cao, Y.; and Sun, A. 2024. SciAgent: Tool-augmented Language Models for Scientific Reasoning. arXiv preprint arXiv:2402.11451.
  25. Miao, N.; Teh, Y. W.; and Rainforth, T. 2023. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. arXiv preprint arXiv:2308.00436.
  26. Mruthyunjaya, V.; Pezeshkpour, P.; Hruschka, E.; and Bhutani, N. 2023. Rethinking language models as symbolic knowledge graphs. arXiv preprint arXiv:2308.13676. — [arXiv:2308.13676](https://arxiv.org/abs/2308.13676) find
  27. Ouyang, S.; Zhang, Z.; Yan, B.; Liu, X.; Han, J.; and Qin, L. 2023. Structured chemistry reasoning with large language models. arXiv preprint arXiv:2311.09656. — [arXiv:2311.09656](https://arxiv.org/abs/2311.09656) find
  28. Petroni, F.; Piktus, A.; Fan, A.; Lewis, P.; Yazdani, M.; De Cao, N.; Thorne, J.; Jernite, Y.; Karpukhin, V.; Maillard, J.; et al. 2020. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252.
  29. Sun, J.; Luo, Y.; Gong, Y.; Lin, C.; Shen, Y.; Guo, J.; and Duan, N. 2023a. Enhancing chain-of-thoughts prompting with iterative bootstrapping in large language models. arXiv preprint arXiv:2304.11657. find
  30. Sun, L.; Han, Y.; Zhao, Z.; Ma, D.; Shen, Z.; Chen, B.; Chen, L.; and Yu, K. 2023b. SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research. arXiv preprint arXiv:2308.13149.
  31. Sun, L.; Han, Y.; Zhao, Z.; Ma, D.; Shen, Z.; Chen, B.; Chen, L.; and Yu, K. 2024. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 19053–19061.
  32. Tyen, G.; Mansoor, H.; Carbune, V.; Chen, P.; and Mak, T. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics ACL 2024, 13894–13908. Bangkok, Thailand and…
  33. Wang, K.; Ren, H.; Zhou, A.; Lu, Z.; Luo, S.; Shi, W.; Zhang, R.; Song, L.; Zhan, M.; and Li, H. 2023. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731.
  34. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824–24837.
  35. Xu, B.; Yang, A.; Lin, J.; Wang, Q.; Zhou, C.; Zhang, Y.; and Mao, Z. 2023. Expertprompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305.14688.
  36. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. find
  37. Yasunaga, M.; Chen, X.; Li, Y.; Pasupat, P.; Leskovec, J.; Liang, P.; Chi, E. H.; and Zhou, D. 2023. Large language models as analogical reasoners. arXiv preprint arXiv:2310.01714.
  38. Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
  39. Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Lu, K.; Tan, C.; Zhou, C.; and Zhou, J. 2024. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models.
  40. Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.
  41. Zhou, A.; Wang, K.; Lu, Z.; Shi, W.; Luo, S.; Qin, Z.; Lu, S.; Jia, A.; Song, L.; Zhan, M.; et al. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jaiswal-2024-improving-physics-reasoning-large-language/