Elamur Research⌕
☰
home / papers / 2024 / GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?

GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?

Paper · 2024 · Zhikai Lei, Tianyi Liang, Hanglei Hu et al.

Summary

LLMs often score well on human-designed benchmarks, but it is unclear whether those scores reflect real, human-like ability or are inflated by data leakage and narrow test coverage. To test this, the authors built GAOKAO-Eval from China's National College Entrance Examination (Gaokao). The exam's questions are written fresh each year under sealed conditions and calibrated on students, and they cover many subjects and formats, including multimodal items. Only models released before the 2024 exam were tested, so the evaluation is effectively closed-book. Experienced teachers graded the open-ended answers. Even with leakage and coverage controlled, scores still did not match human-aligned competence. Using the Rasch model from psychometrics, the authors show two patterns. First, LLM success rates barely change as question difficulty rises. Second, results vary widely among questions of similar difficulty. They also found that teachers disagree when grading LLM answers and that models repeat the same types of errors. Using reasoning-token counts as a difficulty measure, in the style of OpenAI o1, partly closes the gap. The work suggests that difficulty should be defined in LLM terms, not only in human terms.

Key points

  • GAOKAO-Eval contains about 3.95k questions with roughly 0.49k added each year. Questions average 674 tokens in length and come in native Chinese and English. The benchmark builds on Gaokao-Bench and GAOKAO-MM, and its code, model responses and scores are public.
  • Evaluated models were limited to those released before June 6, 2024: Qwen2-72B, Qwen2-57B, Yi-1.5-34B, GLM4-9B, Mixtral 8x22B and VL variants. GPT-4o was included as a reference. More than 54 high-school teachers graded the subjective answers.
  • Results were very uneven across subjects:
    • English: near-perfect scores, with GPT-4o at 142.5/150.
    • Mathematics (Science): weak, with a best score of 89/150 (Qwen2-72B).
    • Physics: the top result was 38/110 (Mixtral 8x22B).
  • The authors trained WQX, a Gaokao-specialised model based on InternLM2-20b-base. It reached 84.94% on GaokaoBench and also improved on Math, MMLU, CMMLU and C-Eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations🎓East China Normal UniversityShanghai · China50 works · 2,174 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhikai Lei, Tianyi Liang, Hanglei Hu, Jin Zhang, Yunhua Zhou, Yunfan Shao, Linyang Li, Chenchui Li, Changbo Wang, Hang Yan and Qipeng Guo. 2024. GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?. arXiv:2412.10056.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.10056 · arXiv 2412.10056

@misc{lei2024gaokao,
  title = {{GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?}},
  author = {Zhikai Lei, Tianyi Liang, Hanglei Hu, Jin Zhang, Yunhua Zhou, Yunfan Shao, Linyang Li, Chenchui Li, Changbo Wang, Hang Yan and Qipeng Guo},
  year = {2024},
  doi = {10.48550/arxiv.2412.10056},
  eprint = {2412.10056},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.10056}
}

References

53 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudo… — Yi: Open Foundation Models by 01.AI
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.
  3. Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on ”a is b” fail to learn ”b is a”. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  4. OpenReview.net, 2024. URL https://openreview.net/forum?id=GPKTIktA0k. ↗
  5. Trevor Bond and Christine Fox. Applying the rasch model: fundamental measurement in the human sciences, 2007. find
  6. William J. Boone and Amity Noltemeyer. Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4(1):1416898, 2017. doi: 10.1080/2331186X.2017.1416898. URL https://www.tandfonline.com/doi/abs/10.1080/2331186X.2017.1416898. — [doi:10.1080/2331186x.2017.1416898](https://doi.org/10.1080/2331186x.2017.1416898) ↗
  7. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjia… — InternLM2 Technical Report
  8. Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024a. URL https://arxiv.org/abs/2405.16473. ↗
  9. Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, and Yue Zhang. See what llms cannot answer: A self-challenge framework for uncovering llm weaknesses. arXiv preprint arXiv:2408.08978, 2024b. — [arXiv:2408.08978](https://arxiv.org/abs/2408.08978) find
  10. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
  11. Tuan Dinh, Jinman Zhao, Samson Tan, Renato M. P. Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. ArXiv, abs/2306.03438, 2023. URL https://api.semanticscholar.org/CorpusID:259089334. — [arXiv:2306.03438](https://arxiv.org/abs/2306.03438) ↗
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — The Llama 3 Herd of Models
  13. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka… — ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
  14. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
  15. OpenReview.net, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ. ↗
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, January 2021b.
  17. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021c.
  18. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing…
  19. Nicolas Hubert, Armelle Brun, and Davy Monticolo. New Ontology and Knowledge Graph for University Curriculum Recommendation. In ISWC 2022 - The 21st International Semantic Web Conference, Hangzhou / Virtual, China, October 2022. URL https://hal.archives-ouvertes.fr/hal-03768154. ↗
  20. Kevin Jesse, Toufique Ahmed, Prem Devanbu, and Emily Morgan. Large language models and simple, stupid bugs. 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp. 563–575, 2023. URL https://api.semanticscholar.org/CorpusID: 257636802. ↗
  21. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning, February 2018. find
  22. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, March 2016.
  23. Myint Swe Khine. Rasch Measurement: Applications in Quantitative Educational Research, volume 1 of Education. Springer Singapore, 1 edition, 2020. ISBN 9789811517900. doi: 10.1007/978-981-15-1800-3. — [doi:10.1007/978-981-15-1800-3](https://doi.org/10.1007/978-981-15-1800-3) find
  24. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2023.
  25. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding in chinese. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Ba…
  26. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin…
  27. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player?, August 2024.
  28. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, October 2022a. find
  29. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning, July 2022b. find
  30. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, ICLR…
  31. OpenReview.net, 2024. URL https://openreview.net/forum?id=KUNzEQMWU7. ↗
  32. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge, September 2019. find
  33. Mistral AI. Mistral Large 2: Designed for Single-Node Inference with Long-Context. https://mistral.ai/news/mistral-large-2407/\#:˜:text=Mistral% 20Large%202%20is%20designed%20for%20single-node%20inference% 20with%20long-context, July 2024. [Online; accessed 2024-09-30]. ↗
  34. Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. Training on the benchmark is not all you need. arXiv preprint arXiv:2409.01790, 2024. — [arXiv:2409.01790](https://arxiv.org/abs/2409.01790) find
  35. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  36. Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024. find
  37. Georg Rasch. Probabilistic models for some intelligence and attainment tests. ERIC, 1993. find
  38. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020… find
  39. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6399. URL https://doi.org/10.1609/aaai.v34i05.6399. — [doi:10.1609/aaai.v34i05.6399](https://doi.org/10.1609/aaai.v34i05.6399) ↗
  40. Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294, 2024.
  41. Florian Tambon, Amin Nikanjam, Foutse Khomh, and Giuliano Antoniol. Assessing programming task difficulty for efficient evaluation of large language models. arXiv preprint arXiv:2407.21227, 2024. — [arXiv:2407.21227](https://arxiv.org/abs/2407.21227) find
  42. InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM, 2023. ↗
  43. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024…
  44. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In North American Chapter of the Association for Computational Li… find
  45. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… find
  46. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… — MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
  47. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluı́s Màrquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy,… find
  48. Xiaotian Zhang, Chun yan Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. ArXiv, abs/2305.12474, 2023. URL https://api.semanticscholar.org/CorpusID:258833562. ↗
  49. Li Zhong and Zilong Wang. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In AAAI Conference on Artificial Intelligence, 2023. URL https://api.semanticscholar.org/CorpusID:261048682. ↗
  50. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023. find
  51. Lexin Zhou, Wout Schellaert, Fernando Martı́nez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, September 2024. ISSN 1476-4687. doi: 10.1038/s41586-024-07930-y. URL https://doi. org/10.1038/s41586-024-079… — [doi:10.1038/s41586-024-07930-y](https://doi.org/10.1038/s41586-024-07930-y) find
  52. Yi Zong and Xipeng Qiu. Gaokao-mm: A chinese human-level benchmark for multimodal models evaluation, 2024.
  53. We provide more details of the proposed method and additional experimental results to help better understand our paper. In summary, this appendix includes the following contents: find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lei-2024-gaokao-eval-high-scores/