Elamur Research⌕
☰
home / papers / 2024 / SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

Paper · 2024 · Kehua Feng, Xinyi Shen, Weijie Wang et al.

Summary

SciKnowEval is a benchmark for measuring how much scientific knowledge large language models hold and how well they use it. The authors argue that earlier science benchmarks either stop at school-level questions, lack a structured evaluation scheme, or ignore safety. Their framework is inspired by a Confucian model of learning and has five levels: memory, comprehension, reasoning, discernment (safe and ethical judgement), and application. The dataset has about 28K questions with solutions across biology, chemistry, physics, and materials science, organised into 58 tasks. The questions were built in three ways:

  • LLM-generated QAs drawn from papers and textbooks
  • LLM-rewritten items taken from existing benchmarks, to reduce contamination
  • Conventional scientific databases converted into text questions

The authors evaluated 20 proprietary, general open-source, and science-specialised models. Proprietary reasoning models came out on top, but every model had trouble with scientific reasoning and real-world application. The benchmark matters because it tests depth and safety, not only recall.

Key points

  • Question sources and formats. Literature came from BioRxiv, PubMed, and LibreTexts. Refactored items came from MedMCQA, SciEval, MMLU, XieZhi, PubMedQA, and HarmfulQA. Formats include multiple choice, true/false, relation extraction, and open content generation, with some answers scored by GPT-4o.
  • Top models. o4-mini ranked first in biology (0.6268), materials (0.7364), and physics (0.8922). o3-mini led chemistry (0.6861).
  • Best open-source models. QwQ-32B was strongest in chemistry, materials, and physics, and DeepSeek-V3 was strongest in biology.
  • Science-specialised models. ChemDFM-13B, ChemLLM-20B-Chat, MolInst-Llama3-8B, SciGLM-6B, and LlaSMol-Mistral-7B all placed 15th–20th in every domain. Even so, MolInst-Llama3-8B beat its base model, Llama3-8B-Instruct, on most application and molecular tasks.
  • GPT-4o by level. It scored above 85% on most L1 and L2 tasks. It struggled at L3 and L5, especially with SMILES strings and protein sequences, and at L4 it often failed to refuse harmful requests.
  • Version inconsistency. The conclusion of this version still refers to 70K problems and 26 models, which conflicts with the abstract (28K questions, 20 models). This summary uses the abstract's figures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Kehua Feng, Xinyi Shen, Weijie Wang, Xiang Zhuang, Yuqi Tang, Qiang Zhang and Keyan Ding. 2024. SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models. arXiv:2406.09098.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.09098 · arXiv 2406.09098

@misc{feng2024sciknoweval,
  title = {{SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models}},
  author = {Kehua Feng, Xinyi Shen, Weijie Wang, Xiang Zhuang, Yuqi Tang, Qiang Zhang and Keyan Ding},
  year = {2024},
  doi = {10.48550/arxiv.2406.09098},
  eprint = {2406.09098},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.09098}
}

References

42 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Anthropic. The Claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. find
  2. R. Bhardwaj and S. Poria. Red-teaming large language models using chain of utterances for safety-alignment. arXiv:2308.09662, 2023. find
  3. H. Cai, X. Cai, J. Chang, S. Li, L. Yao, C. Wang, Z. Gao, Y. Li, M. Lin, S. Yang, et al. SciAssess: Benchmarking LLM proficiency in scientific literature analysis. arXiv:2403.01976, 2024. find
  4. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457, 2018.
  5. U. Consortium. UniProt: the universal protein knowledgebase in 2023. Nucleic acids research, 51(D1):D523–D531, 2023. find
  6. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv:2407.21783, 2024.
  7. Y. Fang, X. Liang, N. Zhang, K. Liu, R. Huang, Z. Chen, X. Fan, and H. Chen. Mol-Instructions: A large-scale biomolecular instruction dataset for large language models. arXiv:2306.08018, 2023. find
  8. Z. Gu, X. Zhu, H. Ye, L. Zhang, J. Wang, Y. Zhu, S. Jiang, Z. Xiong, Z. Li, W. Wu, Q. He, R. Xu, W. Huang, J. Liu, Z. Wang, S. Wang, W. Zheng, H. Feng, and Y. Xiao. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation. arXiv:2306.05783, 2023.
  9. M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Helyar, R. Dias, A. Vallone, H. Ren, J. Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024.
  10. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  11. T. Guo, B. Nan, Z. Liang, Z. Guo, N. Chawla, O. Wiest, X. Zhang, et al. What can large language models do in chemistry? A comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662–59688, 2023. find
  12. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv:2009.03300, 2020.
  13. Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu. PubMedQA: A dataset for biomedical research question answering. arXiv:1909.06146, 2019.
  14. W. Jin, C. Coley, R. Barzilay, and T. Jaakkola. Predicting organic reaction outcomes with Weisfeiler-Lehman network. Advances in neural information processing systems, 30, 2017. find
  15. S. Kim, J. Chen, T. Cheng, A. Gindulyte, J. He, S. He, Q. Li, B. A. Shoemaker, P. A. Thiessen, B. Yu, et al. Pubchem in 2021: new data content and improved web interfaces. Nucleic acids research, 49(D1):D1388–D1395, 2021. find
  16. S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.
  17. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient Memory Management for Large Language Model Serving with Pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, page 611–626. Association for Computing Machinery, 20…
  18. H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring Massive Multitask Language Understanding in Chinese. arXiv:2306.09212, 2023.
  19. A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. find
  20. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. find
  21. S. Ma, B. Zhang, L. M. LaFave, A. S. Earl, Z. Chiang, Y. Hu, J. Ding, A. Brack, V. K. Kartha, T. Tay, et al. Chromatin potential identified by shared single-cell profiling of rna and chromatin. Cell, 183(4):1103–1116, 2020. find
  22. A. Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025, 2025. ↗
  23. OpenAI. Introducing o3 and o4-mini, 2024. find
  24. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pages 27730–27744, 2022.
  25. A. Pal, L. K. Umapathi, and M. Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022. — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  26. QWQ-32B Team. Qwq-32b: A powerful and open large language model, 2024. find
  27. L. Sun, Y. Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061, 2024. find
  28. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science. arXiv:2211.09085, 2022.
  29. X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang. SciBench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv:2307.10635, 2023.
  30. J. Welbl, N. F. Liu, and M. Gardner. Crowdsourcing multiple choice science questions. arXiv:1707.06209, 2017.
  31. Wikipedia contributors. Doctrine of the mean — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=Doctrine_of_the_Mean&oldid=1295487466, 2025. [Online; accessed 10-August-2025]. ↗
  32. L. Wu, B. Yan, J. Han, R. Li, J. Xiao, S. He, and X. Bo. TOXRIC: A comprehensive database of toxicological data and benchmarks. Nucleic Acids Research, 51(D1):D1432–D1445, 2023. find
  33. Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande. MoleculeNet: A benchmark for molecular machine learning. Chemical science, 9(2):513–530, 2018. find
  34. M. Xu, Z. Zhang, J. Lu, Z. Zhu, Y. Zhang, M. Chang, R. Liu, and J. Tang. PEER: A comprehensive and multi-task benchmark for protein sequence understanding. Advances in Neural Information Processing Systems, 35:35156–35173, 2022. find
  35. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv:2505.09388, 2025.
  36. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. Qwen2 technical report. arXiv:2407.10671, 2024. find
  37. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv:2412.15115, 2024. find
  38. B. Yu, F. N. Baker, Z. Chen, X. Ning, and H. Sun. LlaSMol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. arXiv:2402.09391, 2024. find
  39. D. Zhang, Z. Hu, S. Zhoubian, Z. Du, K. Yang, Z. Wang, Y. Yue, Y. Dong, and J. Tang. SciGLM: Training scientific language models with self-reflective instruction annotation and tuning. arXiv:2401.07950, 2024. find
  40. D. Zhang, W. Liu, Q. Tan, J. Chen, H. Yan, Y. Yan, J. Li, W. Huang, X. Yue, D. Zhou, et al. ChemLLM: A chemical large language model. arXiv:2402.06852, 2024. find
  41. Z. Zhao, D. Ma, L. Chen, L. Sun, Z. Li, H. Xu, Z. Zhu, S. Zhu, S. Fan, G. Shen, et al. ChemDFM: Dialogue foundation model for chemistry. arXiv:2401.14818, 2024. — [arXiv:2401.14818](https://arxiv.org/abs/2401.14818) find
  42. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. arXiv:2304.06364, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2024-sciknoweval-evaluating-multi-level-scientific/