Elamur Research
☰
home / papers / 2023 / SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models

SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2307.10635

Summary

Benchmarks for scientific reasoning in LLMs have mostly used school-level questions that need only simple arithmetic, and they rarely include figures. That makes models look stronger at science than they are. SciBench fills this gap with a curated set of college-level problems in mathematics, chemistry and physics, plus exam questions from computer science and mathematics. It includes both text-only and multimodal items. The authors test open-source and proprietary LLMs under several prompting setups: zero-shot, few-shot, Chain-of-Thought (CoT) and use of external tools. The best configuration reaches only 43.22% overall. To find out why models fail, the authors define ten problem-solving skills and attribute each error to a missing skill. GPT-3.5 does this labeling through a self-critique protocol, and human annotators then check its decisions. No prompting strategy is clearly best. Each one strengthens some skills and weakens others. The benchmark and its error analysis give a harder and more detailed way to measure scientific reasoning. Dataset and code are public.

Key points

  • SciBench covers college-level mathematics, chemistry and physics problems, plus CS and math exam questions, with text-only and multimodal settings. It was published at ICML 2024 and is hosted at scibench-ucla.github.io.
  • The best model/prompting combination scores only 43.22% overall.
  • In error attribution, GPT-3.5 labels which of ten skills each wrong answer lacked. Human review then removes about 20% of these labels as misclassified.
  • Zero-shot CoT cuts calculation errors from 29.0% to 13.6%. It also increases causal-reasoning errors (18.3% → 32.2%) and logical-decomposition errors (18.3% → 25.4%).
  • External tools reduce calculation errors from 14.5% (few-shot CoT) to 6.2%. They hurt code-conversion skills, meaning the model is worse at writing correct programs for the solution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

129 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 120 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.10635 · arXiv 2307.10635

@misc{wang2023scibench,
  title = {{SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models}},
  author = {Wang, Xiaoxuan and Hu, Ziniu and Lu, Pan and Zhu, Yanqiao and Zhang, Jieyu and Subramaniam, Satyen and Loomba, Arjun R. and Zhang, Shichang and Sun, Yizhou and Wang, Wei},
  year = {2023},
  doi = {10.48550/arxiv.2307.10635},
  eprint = {2307.10635},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.10635}
}

References

62 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Claude2. https://www.anthropic.com/index/claude-2, 2023. 5 ↗
  2. Arora, D., Singh, H. G., et al. Have llms advanced enough? a challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074, 2023. 1, 3 find
  3. Atkins, P., Atkins, P. W., and de Paula, J. Atkins’ physical chemistry. Oxford university press, 2014a. 4, 12 find
  4. Atkins, P., De Paula, J., and Friedman, R. Physical chemistry: quanta, matter, and change. Oxford University Press, USA, 2014b. 2, 4, 12 find
  5. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 7 find our card
  6. Boyce, W. E., DiPrima, R. C., and Meade, D. B. Elementary differential equations and boundary value problems. John Wiley & Sons, 2021. 4, 12 find
  7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1 find
  8. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. 4 find our card
  9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 1 find our card
  10. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research (TMLR), 2023a. 1, 5, 7 find our card
  11. Chen, W., Yin, M., Ku, M., Lu, P., Wan, E., Ma, X., Xu, J., Xia, T., and Wang, X. Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524, 2023b. 3, 5 find our card
  12. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 1, 3, 4 find our card
  13. Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., Zhang, W., Li, Y., Yan, H., Gao, Y., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang, X., Qiao, Y., Lin, D., and Wang, J. Internlm-xcomposer2: Mastering free-form text-image composition and compre… find
  14. Engel, T. and Reid, P. J. Thermodynamics, statistical thermodynamics, and kinetics. Prentice Hall Upper saddle River, 2010. 4, 12 find
  15. Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867, 2023. 4 find our card
  16. Fu, Y., Ou, L., Chen, M., Wan, Y., Peng, H., and Khot, T. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance. arXiv preprint arXiv:2305.17306, 2023. 3 find
  17. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. 1, 5 find
  18. Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., Li, H., and Qiao, Y. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023. 1 find
  19. Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., and Jacobsen, H.-A. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pp. 1197–1208, 2013. 3 find
  20. Google. Bard. https://bard.google.com, 2023. 7 ↗
  21. Guo, T., Guo, K., Liang, Z., Guo, Z., Chawla, N. V., Wiest, O., Zhang, X., et al. What indeed can gpt models do in chemistry? a comprehensive benchmark on eight tasks. arXiv preprint arXiv:2305.18365, 2023. 3 — [arXiv:2305.18365](https://arxiv.org/abs/2305.18365) find
  22. Halliday, D., Resnick, R., and Walker, J. Fundamentals of physics. John Wiley & Sons, 2013. 4, 12 find
  23. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 1, 3 find our card
  24. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 1, 3, 4 find our card
  25. Hogg, R. V., Tanis, E. A., and Zimmerman, D. L. Probability and statistical inference, volume 993. Macmillan New York, 1977. 4, 13 find
  26. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022. 1 find our card
  27. JaidedAI. Easyocr: Ready-to-use ocr. https://github.com/J aidedAI/EasyOCR, 2022. 7 ↗
  28. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 5 find
  29. Kabir, S., Udo-Imeh, D. N., Kou, B., and Zhang, T. Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions. arXiv preprint arXiv:2308.02312, 2023. 4 — [arXiv:2308.02312](https://arxiv.org/abs/2308.02312) find
  30. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. 1 find our card
  31. Levine, I. N., Busch, D. H., and Shull, H. Quantum chemistry, volume 6. Pearson Prentice Hall Upper Saddle River, NJ, 2009. 4, 12 find
  32. Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. 7 — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) find
  33. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023a. 7 find our card
  34. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023b. 1 find our card
  35. Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., and Zhang, Y. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439, 2023c. 4 find our card
  36. Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joi… find our card
  37. Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021b. 3 find
  38. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. 1, 3, 5 find
  39. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023a. 3 find our card
  40. Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023b. 1 find our card
  41. Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023c. 3 find our card
  42. Lu, P., Qiu, L., Yu, W., Welleck, S., and Chang, K.-W. A survey of deep learning for mathematical reasoning. In The 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023d. 3 find
  43. McQuarrie, D. A. Quantum chemistry. University Science Books, 2008. 4, 12 find
  44. Mishra, S., Finlayson, M., Lu, P., Tang, L., Welleck, S., Baral, C., Rajpurohit, T., Tafjord, O., Sabharwal, A., Clark, P., et al. Lila: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. 3 find our card
  45. OpenAI. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/., 2022. 1, 5 ↗
  46. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 5, 7 find our card
  47. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. 1 find our card
  48. Stewart, J., Watson, S., and Clegg, D. Calculus: Early transcendentals, 8th. Edition, Brooks/Cole, Cengae learning, 2012. 4, 13 find
  49. Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., and Yu, K. Scieval: A multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308.13149, 2023. 3 find
  50. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. 3 find our card
  51. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. 3 find our card
  52. Thornton, S. T. and Marion, J. B. Classical dynamics of particles and systems. Cengage Learning, 2021. 4, 12 find
  53. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. 1 find
  54. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. 5 find
  55. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 1 find our card
  56. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. 1, 7, 8 find our card
  57. Welleck, S., Liu, J., Bras, R. L., Hajishirzi, H., Choi, Y., and Cho, K. Naturalproofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021. 3 find
  58. Zhang, D., Hu, Z., Zhoubian, S., Du, Z., Yang, K., Wang, Z., Yue, Y., Dong, Y., and Tang, J. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. arXiv preprint arXiv:2401.07950, 2024. 4 find
  59. Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023a. 1 find our card
  60. Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023b. 1 find our card
  61. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023. 1, 3, 7 find our card
  62. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. 1 find our card