Elamur Research⌕
☰
home / papers / 2023 / Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

Paper · 2023 · Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye et al.

Summary

Benchmarks such as MMLU, BIG-bench and HELM quickly become too easy for modern LLMs and may leak into their training data. Xiezhi addresses this with a very broad, regularly refreshed test of domain knowledge. Its subject structure follows the Chinese Disciplinary Taxonomy of the Ministry of Education. The authors hand-labelled a seed set of graduate entrance exam questions and used it to train an annotation model. That model then assigned discipline tags to a much larger pool of exam questions and to questions generated automatically from academic surveys, so the benchmark can keep growing with fresh material.

Evaluation also changes. Each question gets 50 candidate answers instead of the usual 4, and the model's generation probabilities rank those candidates. Scores are reported with MRR and Hit@k. Across 47 LLMs, the strongest models beat average human practitioners in science, engineering, agronomy and medicine. Humans still lead clearly in economics, law, education, literature, history and management. Xiezhi separates models of different strength better than earlier benchmarks.

Key points

  • Scale and structure: 249,587 multiple-choice questions over 516 disciplines, organised as 13 top-level categories, 118 sub-disciplines and 385 fine-grained fields.
  • Data pipeline:
    • Xiezhi-Meta has 20,124 manually annotated questions from the Chinese Graduate Entrance Examination.
    • It trains the labelling model, which tags about 170k exam questions and about 80k survey-generated questions.
  • Balanced bilingual subsets: Xiezhi-Specialty (14,041 single-domain questions) and Xiezhi-Interdiscipline (10,746 multi-domain questions) come in Chinese and English and are less China-centric.
  • Experimental setup: 47 LLMs were tested, from Bloomz-560m to GPT-4, in 0-, 1- and 3-shot settings on four benchmarks. Xiezhi produced the largest score variance between models.
  • Findings:
    • More parameters do not always help; Pythia-1.4b beat larger Pythia models.
    • Domain-tuned small models such as DoctorGLM and Baize-Healthcare improved in medicine but lost general ability.
  • Availability: code and data are released at github.com/MikeGu721/XiezhiBenchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓East China University of Science and TechnologyShanghai · China14 works · 87 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, Qianyu He, Rui Xu, Wenhao Huang, Jingping Liu, Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng and Yanghua Xiao. 2023. Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation. arXiv:2306.05783.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.05783 · arXiv 2306.05783

@misc{gu2023xiezhi,
  title = {{Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation}},
  author = {Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, Qianyu He, Rui Xu, Wenhao Huang, Jingping Liu, Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng and Yanghua Xiao},
  year = {2023},
  doi = {10.48550/arxiv.2306.05783},
  eprint = {2306.05783},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.05783}
}

References

47 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aribandi, V.; Tay, Y.; Schuster, T.; Rao, J.; Zheng, H. S.; Mehta, S. V.; Zhuang, H.; Tran, V. Q.; Bahri, D.; Ni, J.; et al. 2021. Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint arXiv:2111.10952.
  2. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022. Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073.
  3. Biderman, S.; Schoelkopf, H.; Anthony, Q.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; Skowron, A.; Sutawika, L.; and van der Wal, O. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. arXiv:2304.01373. find
  4. Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 7432–7439.
  5. Black, S.; Biderman, S.; Hallahan, E.; Anthony, Q.; Gao, L.; Golding, L.; He, H.; Leahy, C.; McDonell, K.; Phang, J.; Pieler, M.; Prashanth, U. S.; Purohit, S.; Reynolds, L.; Tow, J.; Wang, B.; and Weinbach, S. 2022. GPT-NeoX-20B: An Open-Source Autoregressive Language Model.
  6. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  7. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. find
  8. Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, E.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Conover, M.; Hayes, M.; Mathur, A.; Meng, X.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tu… ↗
  10. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335.
  11. Fu, Y.; Peng, H.; and Khot, T. 2022. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. find
  12. Yao Fu’s Notion. Fu, Y.; Peng, H.; Ou, L.; Sabharwal, A.; and Khot, T. 2023. Specializing Smaller Language Models towards Multi-Step Reasoning. arXiv preprint arXiv:2301.12726. H2O.ai. 2023. h2oGPT - The world’s best open source GPT. https://github.com/h2oai/h2ogpt. ↗
  13. Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938.
  14. Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35: 30016–30030. find
  15. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  16. Hu, Z.; Lan, Y.; Wang, L.; Xu, W.; Lim, E.-P.; Lee, R. K.-W.; Bing, L.; and Poria, S. 2023. LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. arXiv preprint arXiv:2304.01933.
  17. Huang, L.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277.
  18. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; et al. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. arXiv preprint arXiv:2305.08322.
  19. Ji, Y.; Deng, Y.; Gong, Y.; Peng, Y.; Niu, Q.; Zhang, L.; Ma, B.; and Li, X. 2023a. Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases. arXiv preprint arXiv:2303.14742.
  20. Ji, Y.; Gong, Y.; Deng, Y.; Peng, Y.; Niu, Q.; Ma, B.; and Li, X. 2023b. Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation. arXiv preprint arXiv:2304.07854. — [arXiv:2304.07854](https://arxiv.org/abs/2304.07854) find
  21. Kpf, A.; Kilcher, Y.; von Rütte, D.; Anagnostidis, S.; Tam, Z.-R.; Stevens, K.; Barhoum, A.; Duc, N. M.; Stanley, O.; Nagyfi, R.; ES, S.; Suri, S.; Glushkov, D.; Dantuluri, A.; Maguire, A.; Schuhmann, C.; Nguyen, H.; and Mattick, A. 2023. OpenAssistant Conversations – Democratizing Large Language Mo…
  22. Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2023. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212.
  23. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. find
  24. Liu, C.; Jin, R.; Ren, Y.; Yu, L.; Dong, T.; Peng, X.; Zhang, S.; Peng, J.; Zhang, P.; Lyu, Q.; et al. 2023. M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models. arXiv preprint arXiv:2305.10263.
  25. Muennighoff, N.; Wang, T.; Sutawika, L.; Roberts, A.; Biderman, S.; Scao, T. L.; Bari, M. S.; Shen, S.; Yong, Z.-X.; Schoelkopf, H.; Tang, X.; Radev, D.; Aji, A. F.; Almubarak, K.; Albanie, S.; Alyafeai, Z.; Webson, A.; Raff, E.; and Raffel, C. 2022. Crosslingual Generalization through Multitask Fin…
  26. OpenAI. 2023a. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt. ↗
  27. OpenAI. 2023b. GPT-4 Technical Report. arXiv:2303.08774.
  28. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744. Perez, E.; Ringer, S.; Lukošiūt…
  29. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1): 5485–5551.
  30. Sai, A. B.; Mohankumar, A. K.; and Khapra, M. M. 2022. A survey of evaluation metrics used for NLG systems. ACM Computing Surveys (CSUR), 55(2): 1–39. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; Gallé, M.; et al. 2022. Bloom: A 176… find
  31. Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  32. StabilityAI. 2023. StableLM: Stability AI Language Models. https://github.com/Stability-AI/StableLM. ↗
  33. Sun, T.; Xiaotian, Z.; Zhengfu, H.; Peng, L.; Qinyuan, C.; Hang, Y.; Xiangyang, L.; Yunfan, S.; Qiong, T.; Xingjian, Z.; Ke, C.; Yining, Z.; Zhejian, Z.; Ruixiao, L.; Jun, Z.; Yunhua, Z.; Linyang, L.; Xiaogui, Y.; Lingling, W.; Zhangyue, Y.; Xuanjing, H.; and Xipeng, Q. 2023a. FudanNLP Moss. find
  34. Sun, T.; Zhang, X.; He, Z.; Li, P.; Cheng, Q.; Yan, H.; Liu, X.; Shao, Y.; Tang, Q.; Zhao, X.; Chen, K.; Zheng, Y.; Zhou, Z.; Li, R.; Zhan, J.; Zhou, Y.; Li, L.; Yang, X.; Wu, L.; Yin, Z.; Huang, X.; and Qiu, X. 2023b. MOSS: An open-source tool-augmented conversational language model from Fudan Univ…
  35. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford alpaca. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière… ↗
  36. Wang, E. J.; Alexiuk, C.; Bo, Y.; Yang, Z.; Kwok, K.; Gusev, I.; Echavez, A.; et al. 2023a. Alpaca-LoRA. https://github.com/tloen/alpaca-lora. ↗
  37. Wang, J.; Liang, Y.; Meng, F.; Shi, H.; Li, Z.; Xu, J.; Qu, J.; and Zhou, J. 2023b. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048.
  38. Wei, J.; Tay, Y.; Bommasani, R.; Raffel, C.; Zoph, B.; Borgeaud, S.; Yogatama, D.; Bosma, M.; Zhou, D.; Metzler, D.; et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  39. Xiong, H.; Wang, S.; Zhu, Y.; Zhao, Z.; Liu, Y.; Huang, L.; Wang, Q.; and Shen, D. 2023. DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task. arXiv:2304.01097. find
  40. Xu, C.; Guo, D.; Duan, N.; and McAuley, J. 2023. Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data. arXiv preprint arXiv:2304.01196.
  41. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  42. Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; Tam, W. L.; Ma, Z.; Xue, Y.; Zhai, J.; Chen, W.; Liu, Z.; Zhang, P.; Dong, Y.; and Tang, J. 2023. GLM-130B: An Open Bilingual Pre-trained Model. In The Eleventh International Conference on Learning Represen…
  43. Zeng, H. 2023. Measuring Massive Multitask Chinese Understanding. arXiv preprint arXiv:2304.12986. find
  44. Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  45. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. arXiv preprint arXiv:2304.06364.
  46. Zhou, C.; Li, Q.; Li, C.; Yu, J.; Liu, Y.; Wang, G.; Zhang, K.; Ji, C.; Yan, Q.; He, L.; et al. 2023. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419. find
  47. Zhu, M.; Deng, Z.; Xiong, W.; Yu, M.; Zhang, M.; and Wang, W. Y. 2019. Neural Correction Model for Open-Domain Named Entity Recognition. arXiv preprint arXiv:1909.06058. — [arXiv:1909.06058](https://arxiv.org/abs/1909.06058) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gu-2023-xiezhi-ever-updating-benchmark/