Elamur Research⌕
☰
home / papers / 2023 / SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark

SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark

Paper · 2023 · Liang Xu, Anqi Li, Lei Zhu et al.

Summary

Most LLM benchmarks, including Chinese ones such as C-Eval and MMCU, rely on multiple-choice accuracy. That metric says little about how users experience a model in free-form conversation. SuperCLUE addresses this gap for Chinese with three complementary parts:

  • CArena collects real user queries and pairwise votes from LangYa Leaderboard, an anonymous Elo-based battle platform. Its win-and-tie rates serve as the gold standard.
  • OPEN holds open-ended single- and multi-turn questions across ten capability categories drawn from annotated user queries. GPT-4 scores the answers automatically.
  • CLOSE converts each single-turn OPEN item into a multiple-choice question. GPT-3.5 generates the options, which are then manually verified.

Across 11 LLMs, GPT-4 leads by a wide margin, and MiniMax is the strongest Chinese model. GPT-4's judgments agree well with human ratings. Closed-ended accuracy alone does not significantly track user preference, but combining CLOSE with multi-turn OPEN scores predicts CArena results best. The work suggests that Chinese LLM evaluation needs open-ended, conversational testing alongside exam-style questions.

Key points

  • CArena contains about 9.9k user votes collected since 15 May 2023 on LangYa Leaderboard. The compared models include ChatGLM-130B, ChatGLM2-6B, MOSS, Ziya and MiniMax.
  • The ten capability categories range from semantic understanding and small talk to code generation, calculation, role-playing and safety. OPEN has 30 single-turn and 30 multi-turn questions per category, for 600 items in total.
  • Pearson correlations with CArena are 0.7950 for CLOSE (not significant), 0.8747 for OPEN SINGLE (near significant) and 0.8985 for OPEN MULTIPLE (significant). The best mix, 0.49·CLOSE + 0.51·OPEN MULTIPLE, reaches 0.9397.
  • Scores on CLOSE vary much less across models than scores on OPEN SINGLE. The reported coefficients of variation are roughly 0.11 and 0.34, and the paper's ordering of these values is somewhat ambiguous. This means CLOSE separates models poorly.
  • Wenxin Yiyan, 360 and SparkDesk were evaluated on OPEN and CLOSE but are missing from CArena because platform access was limited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Westlake UniversityHangzhou · China45 works · 8,028 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang and Zhenzhong Lan. 2023. SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark. arXiv:2307.15020.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.15020 · arXiv 2307.15020

@misc{xu2023superclue,
  title = {{SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark}},
  author = {Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang and Zhenzhong Lan},
  year = {2023},
  doi = {10.48550/arxiv.2307.15020},
  eprint = {2307.15020},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.15020}
}

References

32 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Ch… — Language Models are Few-Shot Learners
  2. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. find
  3. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo…
  4. Dolan, W. B.; and Brockett, C. 2005. Automatically Constructing a Corpus of Sentential Paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
  5. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335.
  6. Goyal, T.; Li, J. J.; and Durrett, G. 2023. News Summarization and Evaluation in the Era of GPT-3. arXiv:2209.12356. find
  7. Gu, Z.; Zhu, X.; Ye, H.; Zhang, L.; Wang, J.; Jiang, S.; Xiong, Z.; Li, Z.; He, Q.; Xu, R.; Huang, W.; Wang, Z.; Wang, S.; Zheng, W.; Feng, H.; and Xiao, Y. 2023. Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation. arXiv:2306.05783.
  8. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring Massive Multitask Language Understanding. CoRR, abs/2009.03300.
  9. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; Fu, Y.; Sun, M.; and He, J. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. arXiv preprint arXiv:2305.08322. Kasneci, E.; Seßler, K.; Küchemann, S.; Bannert,…
  10. Lee Rodgers, J.; and Nicewander, W. A. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1): 59–66. find
  11. Lewkowycz, A.; Slone, A.; Andreassen, A.; Freeman, D.; Dyer, E. S.; Mishra, G.; Gur-Ari, G.; Lee, J.; Sohl-dickstein, J.; Chiafullo, K.; Fedus, L. B.; Fiedel, N.; Liu, R.; Misra, V.; and Ramasesh, V. V. 2022. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language model…
  12. Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsu-lab/alpaca eval. ↗
  13. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; Newman, B.; Yuan, B.; Yan, B.; Zhang, C.; Cosgrove, C.; Manning, C. D.; Ré, C.; Acosta-Navas, D.; Hudson, D. A.; Zelikman, E.; Durmus, E.; Ladhak, F.; Rong, F.; Ren, H.; Yao, H.; Wa… find
  14. Nov, O.; Singh, N.; and Mann, D. 2023. Putting ChatGPT’s Medical Advice to the (Turing) Test. arXiv:2301.10035. — [arXiv:2301.10035](https://arxiv.org/abs/2301.10035) find
  15. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
  16. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instruction…
  17. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
  18. Sallam, M. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, volume 11, 887. find
  19. MDPI. Sarlin, P.; DeTone, D.; Malisiewicz, T.; and Rabinovich, A. 2019. SuperGlue: Learning Feature Matching with Graph Neural Networks. CoRR, abs/1911.11763. find
  20. Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 1631–1642.
  21. Spearman, C. 1987. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 100(3/4): 441–471. find
  22. Sun, T.; Zhang, X.; He, Z.; Li, P.; Cheng, Q.; Yan, H.; Liu, X.; Shao, Y.; Tang, Q.; Zhao, X.; Chen, K.; Zheng, Y.; Zhou, Z.; Li, R.; Zhan, J.; Zhou, Y.; Li, L.; Yang, X.; Wu, L.; Yin, Z.; Huang, X.; and Qiu, X. 2023. MOSS: Training Conversational Language Models from Synthetic Data. Touvron, H.; La…
  23. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353– 355. Brussels, Belgium: As…
  24. Wang, J.; Zhang, Y.; Zhang, L.; Yang, P.; Gao, X.; Wu, Z.; Dong, X.; He, J.; Zhuo, J.; Yang, Q.; Huang, Y.; Li, X.; Wu, Y.; Lu, J.; Zhu, X.; Chen, W.; Han, T.; Pan, K.; Wang, R.; Wang, H.; Wu, X.; Zeng, Z.; Chen, C.; Gan, R.; and Zhang, J. 2022. Fengshenbang 1.0: Being the Foundation of Chinese Cogn… — [arXiv:2209.02970](https://arxiv.org/abs/2209.02970) find
  25. Williams, A.; Nangia, N.; and Bowman, S. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 11…
  26. Xu, L.; Hu, H.; Zhang, X.; Li, L.; Cao, C.; Li, Y.; Xu, Y.; Sun, K.; Yu, D.; Yu, C.; Tian, Y.; Dong, Q.; Liu, W.; Shi, B.; Cui, Y.; Li, J.; Zeng, J.; Wang, R.; Xie, W.; Li, Y.; Patterson, Y.; Tian, Z.; Zhang, Y.; Zhou, H.; Liu, S.; Zhao, Z.; Zhao, Q.; Yue, C.; Zhang, X.; Yang, Z.; Richardson, K.; an… — CLUE: A Chinese Language Understanding Evaluation Benchmark
  27. Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  28. Zeng, H. 2023. Measuring Massive Multitask Chinese Understanding. arXiv:2304.12986. find
  29. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068.
  30. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
  31. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A. S. S.; Chen, W.; and Duan, N. 2023. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. ArXiv, abs/2304.06364.
  32. Zhuang, L.; Wayne, L.; Ya, S.; and Jun, Z. 2021. A Robustly Optimized BERT Pre-training Approach with Post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, 1218–1227. Huhhot, China: Chinese Information Processing Society of China.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/