Elamur Research⌕
☰
home / papers / 2023 / Evaluating the Performance of Large Language Models on GAOKAO Benchmark

Evaluating the Performance of Large Language Models on GAOKAO Benchmark

Summary

Popular LLM benchmarks such as MMLU, C-Eval and AGIEval consist almost entirely of objective items. They reveal little about generative ability and give no intuitive, human-comparable sense of model skill. The authors address this gap with GAOKAO-Bench, a dataset of national Chinese college entrance exam (GAOKAO) papers from 2010–2022 covering 9 subjects. The papers were converted from PDF to JSON, with formulas in LaTeX, using scripts plus manual annotation.

Models answer zero-shot, with prompts tailored to each question type that also fix the output format. Objective answers are scored by regular-expression matching. Each subjective answer is graded by two teachers. Results are then mapped onto 750-point totals for the sciences and humanities tracks.

GPT-4 performs best, but every model shows a clear subject imbalance: strong in English, biology and geography, weak in mathematics and physics. The authors also use GPT-4-turbo as a grader guided by teacher-written marking criteria and find fairly close agreement with human scores. The benchmark offers a realistic Chinese testbed and a path toward cheaper grading of subjective answers.

Key points

  • GAOKAO-Bench has 2811 questions: 1781 objective (e.g., 1418 single-choice) and 1030 subjective (786 open-ended, 218 cloze, 26 correction). The resource is released at github.com/OpenLMLab/GAOKAO-Bench.
  • Objective scoring rates: GPT-4-0314 reaches 72.2%, GPT-4-0613 71.6%, ERNIE-Bot-0615 56.6% and GPT-3.5-turbo-0301 53.2%. Open-source models (LLaMA, Vicuna, Baichuan2, ChatGLM) range from about 21% to 44%.
  • On teacher-graded subjective questions, GPT-4-0314 scores 51.9% and ERNIE-Bot-0615 48.4%. Science and humanities mathematics stay below about 28% for all tested models.
  • To check for data leakage, the authors built GAOKAO-Bench-2023. GPT-4 drops by only 0.6–2.4 points on it, while ChatGLM-6b loses 6.7.
  • GPT-4-turbo (GPT-4-1106-preview) was used as a judge. With marking criteria, its scores correlate with human scores at Spearman ≈0.83–0.85 and Kendall-Tau ≈0.69–0.71, compared with about 0.80–0.82 Spearman without criteria. The judge's converted totals deviate from human totals by under 2% of the total in the sciences and about 5% in the humanities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.12474 · arXiv 2305.12474

@misc{zhang2023evaluating,
  title = {{Evaluating the Performance of Large Language Models on GAOKAO Benchmark}},
  author = {Zhang, Xiaotian and Li, Chunyang and Zong, Yi and Ying, Zhengyu and He, Liang and Qiu, Xipeng and Sun, Tianxiang and Li, Peng and Meng, Shiqiao and Zheng, Yanjun and Zhan, Jun and Yin, Zhangyue and Hu, Xiannian and Quan, Guofeng and Wang, Qixiang},
  year = {2023},
  doi = {10.48550/arxiv.2305.12474},
  eprint = {2305.12474},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.12474}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774. find our card
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  3. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. find our card
  5. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. find our card
  6. ArXiv, Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.… find our card
  7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models. find our card
  8. Hongye Tan, Xiaoyue Wang, Yu Ji, Ru Li, Xiaoli Li, Zhiwei Hu, Yunxiao Zhao, and Xiaoqi Han. 2021. Gcrc: A new challenging mrc dataset from gaokao chinese for explainable evaluation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1319–1330. find
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. find our card
  10. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  11. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. find our card
  12. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  13. Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023. Multidimensional evaluation of text summarization with in-context learning. find
  14. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. find our card
  15. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. find our card
  16. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find our card
  17. √ Weizhe Yuan and Pengfei Liu. 2022. restructured pre-training. arXiv preprint arXiv:2206.11147. — [arXiv:2206.11147](https://arxiv.org/abs/2206.11147) find
  18. D. − 13 − 3 3 i Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. Glm-130b: An open bilingual pre-trained model. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-evaluating-performance-large-language/