Elamur Research⌕
☰
home / papers / 2023 / Measuring Massive Multitask Chinese Understanding

Measuring Massive Multitask Chinese Understanding

Paper · 2023 · Hui Zeng

Summary

At the time of this work, Chinese-capable large language models were being released quickly, but no Chinese counterpart to MMLU existed to assess what they knew. Hui Zeng introduces MMCU, a Chinese exam-style benchmark. Professionals manually collected its single- and multiple-answer questions from freely available sources: university medical exams, the national legal qualification exam, psychological counselor and psychology graduate entrance exams, and the Gaokao. Models are scored by accuracy in zero-shot mode and in a five-example few-shot mode.

Seven models were tested: four bloomz variants, ChatGLM 6B, MOSS 16B and GPT-3.5-turbo. GPT-3.5-turbo led every domain but averaged only 0.413 zero-shot. Law was hard for all models. Parameter count did not predict results well, which the author attributes to differences in training methods and data. Few-shot prompting generally lowered scores. The benchmark shows that current models are still far from expert level in specialised Chinese domains.

Key points

  • Dataset size: about 11,900 questions in total.
    • Medicine: 2,819 questions across 15 subtasks.
    • Law: 3,695 questions.
    • Psychology: 2,001 questions.
    • Education: 3,331 questions across 8 Gaokao subjects.
    • The paper reports a 55-question few-shot dev set and a 11,845-question test set.
  • Zero-shot averages: GPT-3.5-turbo 0.413, bloomz_7b1_mt 0.300, ChatGLM 6B 0.282, MOSS 16B 0.208 and bloomz_1b1 0.204. The smaller bloomz_560m (0.227) outperformed MOSS 16B.
  • Best and worst results: the highest single score was GPT-3.5-turbo on clinical medicine (0.693). In law, no model went above 0.239.
  • Education subtasks: GPT-3.5-turbo ranged from 59.9% in biology down to 31.0% in Chinese language. Physics and math were weak for all models.
  • Few-shot effect: few-shot prompting often reduced accuracy. ChatGLM 6B scored lower on every education subtask, which the author links to instruction tuning and alignment. Code and results are at github.com/Felixgithub2017/MMCU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

Cite

Cite as Hui Zeng. 2023. Measuring Massive Multitask Chinese Understanding. arXiv:2304.12986.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.12986 · arXiv 2304.12986

@misc{zeng2023measuring,
  title = {{Measuring Massive Multitask Chinese Understanding}},
  author = {Hui Zeng},
  year = {2023},
  doi = {10.48550/arxiv.2304.12986},
  eprint = {2304.12986},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.12986}
}

References

6 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Greg Brockman, Atty Eleti, Elie Georges, Joanne Jang, Logan Kilpatrick, Rachel Lim, Luke Miller, and Michelle Pokrass. 2023. Introducing ChatGPT and Whisper APIs. https://openai.com/blog/introducing-chatgpt-and-whisper-apis. ↗
  2. Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P…
  3. Tianxiang Sun and Xipeng Qiu. 2023. MOSS . https://txsun1997.github.io/blogs/moss.html ↗
  4. Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  5. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000–6010. bloomz_560m bloomz_1b1 bloomz_3…
  6. Math 0.251 0.233 Physics 0.167 0.185 Chemistry 0.300 0.300 Politics 0.256 0.244 History 0.264 0.256 Geography 0.255 0.248 Biology 0.274 0.278 Average 0.254 0.255 Table 4: Few-shot accuracy of all models on education subtasks 0.229 0.229 0.411 0.252 0.298 0.251 0.281 0.343 0.278 0.427 0.179 0.220 0.2… find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2023-measuring-massive-multitask-chinese-understanding/