Elamur Research⌕
☰
home / papers / 2023 / SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

Paper · 2023 · Liangtai Sun, Yang Han, Zihan Zhao et al.

Summary

Existing LLM benchmarks that touch on science mostly reuse fixed exam-style objective questions. Such questions may already appear in pre-training data, and they do not test open-ended reasoning. Many of these benchmarks also cover only one discipline. SciEval addresses these gaps with an English benchmark of roughly 18,000 questions in chemistry, physics and biology. Drawing on Bloom's taxonomy, it groups questions under four abilities: recalling foundational knowledge, putting that knowledge to use, carrying out scientific computations, and conducting research.

The benchmark has three parts:

  • Static Data: Socratic Q&A posts, converted by GPT-4 into multiple-choice items with manual checking, plus items drawn from public datasets.
  • Dynamic Data: questions regenerated from scientific principles and refreshed periodically to reduce leakage.
  • Experimental Data: subjective questions based on twelve lab experiments.

The authors evaluated many LLMs in answer-only, chain-of-thought and few-shot modes. GPT-4 performed best overall but left considerable headroom. Most models were weak on calculation, and physics was the hardest area. The resource is useful for tracking scientific reasoning in LLMs in a way that is less affected by contamination.

Key points

  • Static Data adds external sources: the USMLE test set of MedQA, 1,000 expert-annotated PubMedQA items recast as judgment questions, and 40% of ChemLLMBench Reagent Selection.
  • On Static Data, only GPT-4, GPT-3.5-turbo and Claude-v1.3 exceeded 60% average accuracy.
  • On the Dynamic physics subset, most models stayed near chance. GPT-3.5-turbo reached 47.19 with CoT, and GPT-4 reached 51.01 with 3-shot prompting.
  • Galactica-30B, trained on scientific text, led on Dynamic counting and calculation questions, outperforming much larger general models.
  • Strong models handled experimental principles and design well but struggled to analyse results. Code and data are available at github.com/OpenDFM/SciEval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen and Kai Yu. 2023. SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research. arXiv:2308.13149.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.13149 · arXiv 2308.13149

@misc{sun2023scieval,
  title = {{SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research}},
  author = {Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen and Kai Yu},
  year = {2023},
  doi = {10.48550/arxiv.2308.13149},
  eprint = {2308.13149},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.13149}
}

References

29 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Blanco-Gonzalez, A.; Cabezon, A.; Seco-Gonzalez, A.; Conde-Torres, D.; Antelo-Riveiro, P.; Pineiro, A.; and Garcia-Fandino, R. 2023. The role of ai in drug discovery: challenges, opportunities, and strategies. Pharmaceuticals, 16(6): 891. find
  2. Chang, Y.; Wang, X.; Wang, J.; Wu, Y.; Zhu, K.; Chen, H.; Yang, L.; Yi, X.; Wang, C.; Wang, Y.; et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109.
  3. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335.
  4. Forehand, M. 2010. Blooms taxonomy. Emerging perspectives on learning, teaching, and technology, 41(4): 47–56. find
  5. Frey, N.; Soklaski, R.; Axelrod, S.; Samsi, S.; Gomez-Bombarelli, R.; Coley, C.; and Gadepally, V. 2022. Neural scaling of deep chemical models. find
  6. Guo, T.; Guo, K.; Liang, Z.; Guo, Z.; Chawla, N. V.; Wiest, O.; Zhang, X.; et al. 2023. What indeed can GPT models do in chemistry? A comprehensive benchmark on eight tasks. arXiv preprint arXiv:2305.18365. — [arXiv:2305.18365](https://arxiv.org/abs/2305.18365) find
  7. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  8. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  9. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; et al. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.
  10. Jin, D.; Pan, E.; Oufattole, N.; Weng, W.-H.; Fang, H.; and Szolovits, P. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14): 6421.
  11. Jin, Q.; Dhingra, B.; Liu, Z.; Cohen, W. W.; and Lu, X. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  12. Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199–22213.
  13. Krathwohl, D. R. 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice, 41(4): 212–218. find
  14. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. find
  15. Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2521. find
  16. Luo, R.; Sun, L.; Xia, Y.; Qin, T.; Zhang, S.; Poon, H.; and Liu, T.-Y. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6): bbac409. find
  17. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
  18. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744.
  19. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318.
  20. Schulman, J.; Zoph, B.; Kim, C.; Hilton, J.; Menick, J.; Weng, J.; Uribe, J. F. C.; Fedus, L.; Metz, L.; Pokorny, M.; et al. 2022. ChatGPT: Optimizing language models for dialogue. OpenAI blog. find
  21. Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; Chung, H. W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S.; et al. 2023. Large language models encode clinical knowledge. Nature, 1–9.
  22. Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  23. Sun, T.; Zhang, X.; He, Z.; Li, P.; Cheng, Q.; Yan, H.; Liu, X.; Shao, Y.; Tang, Q.; Zhao, X.; Chen, K.; Zheng, Y.; Zhou, Z.; Li, R.; Zhan, J.; Zhou, Y.; Li, L.; Yang, X.; Wu, L.; Yin, Z.; Huang, X.; and Qiu, X. 2023. MOSS: Training Conversational Language Models from Synthetic Data. find
  24. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford alpaca: An instruction-following llama model. find
  25. Taylor, R.; Kardas, M.; Cucurull, G.; Scialom, T.; Hartshorn, A.; Saravia, E.; Poulton, A.; Kerkez, V.; and Stojnic, R. 2022. GALACTICA: A Large Language Model for Science. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.…
  26. WANG, F.; and MIAO, Q. 2023. Novel Paradigm for AI-driven Scientific Research: From AI4S to Intelligent Science. Bulletin of Chinese Academy of Sciences (Chinese Version), 38(4): 536–540. find
  27. Wang, X.; Hu, Z.; Lu, P.; Zhu, Y.; Zhang, J.; Subramaniam, S.; Loomba, A. R.; Zhang, S.; Sun, Y.; and Wang, W. 2023. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. arXiv preprint arXiv:2307.10635.
  28. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685.
  29. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2023-scieval-multi-level-large-language/