Elamur Research⌕
☰
home / papers / 2023 / CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

Paper · 2023 · Tianwen Wei, Jian Luan, Wei Liu et al.

Summary

Existing math benchmarks for large language models are mostly English, often pitched at college level, and produce scores that are hard for non-specialists to interpret. The authors, from Xiaomi AI Lab, introduce CMATH, a test-only collection of about 1.7k Chinese elementary school word problems taken from freely available workbooks and exams. Problems with images were removed, and the remaining text was deduplicated, cleaned and checked by hand. Each item carries a school grade (1–6), a numeric answer, a manually counted number of reasoning steps and an automatically computed digit count. With these labels, results can be reported in school terms, for example the highest grade a model reliably passes. The authors ran ten Chinese-capable chat and foundation models zero-shot, and only GPT-4 reached the 60% pass mark in every grade. A second test added irrelevant numeric details to problems. Every model except GPT-4 lost much of its accuracy. CMATH gives an easy-to-read measure of arithmetic and reasoning ability, and of robustness, for Chinese LLMs.

Key points

  • Dataset size by grade: grades 1–6 contain 254, 353, 337, 220, 227 and 298 problems. Average length, step count and digit count generally rise with grade (steps go from 1.3 to 3.0).
  • Models tested: GPT-4, ChatGPT (gpt-3.5-turbo), Chinese-Alpaca-13B/33B, MOSS-16B, Ziya-LLaMA-13B, RWKV-7B, Baichuan-7B, ChatGLM-6B and ChatGLM2-6B. All were evaluated zero-shot, with no extra prompting.
  • Main result: GPT-4 was the only model to reach at least 60% accuracy across all six grades. The other models failed at various grade levels.
  • Distractor set: a separate set of 60 problems (10 per grade), each extended with 1–5 inserted irrelevant details containing one number apiece. ChatGPT lost roughly 30% accuracy with only two distractors, while GPT-4 declined only slightly.
  • Advantage over earlier Chinese datasets: Math23K and APE210K lack fine-grained labels such as grade or step count. AGI-Eval and C-Eval contain much harder material, ranging from middle school to college level.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksMathematics · 140 worksApplied Mathematics · 21 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong and Bin Wang. 2023. CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?. arXiv:2306.16636.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.16636 · arXiv 2306.16636

@misc{wei2023cmath,
  title = {{CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?}},
  author = {Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong and Bin Wang},
  year = {2023},
  doi = {10.48550/arxiv.2306.16636},
  eprint = {2306.16636},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.16636}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
  2. Baichuan-7B. https://github.com/baichuan-inc/baichuan-7B/blob/ main/README_EN.md . Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.… ↗
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems.
  4. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177.
  5. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: general language model pretraining with autoregressive blank infilling. pages 320–335.
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding.
  7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset.
  8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models.
  9. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.
  10. IDEA-CCNL. 2023. Idea-ccnl/ziya-llama-13b-v1. https://huggingface.co/IDEA-CCNL/ Ziya-LLaMA-13B-v1/blob/main/README.md . ↗
  11. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. find
  12. OpenAI. 2023. GPT-4 technical report.
  13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  14. RWKV-4-raven. https://huggingface.co/BlinkDL/rwkv-4-raven . ↗
  15. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito,… find
  16. Tianxiang Sun and Xipeng Qiu. 2023. MOSS. Github. https://github.com/OpenLMLab/MOSS/ blob/main/README_en.md . ↗
  17. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science.
  18. THUDM. 2023a. ChatGLM-6B. https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md . ↗
  19. ChatGLM2-6B. https://github.com/THUDM/ChatGLM2-6B/blob/main/ README_EN.md . ↗
  20. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845–854, Copenhagen, Denmark. Association for Computational Linguistics. find
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
  22. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. GLM-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  23. Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. find
  24. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2023-cmath-language-model-pass/