Elamur Research⌕
☰
home / papers / 2024 / SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese

SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese

Paper · 2024 · Liang Xu, Hang Xue, Lei Zhu et al.

Summary

Popular math benchmarks such as GSM8K are English-only and do not probe multi-step or sustained inference well. To address this for Chinese LLMs, the SuperCLUE team built SuperCLUE-Math6 (SC-Math6). It contains 1072 grade-school word problems, drawn from Chinese exams and textbooks and manually rewritten to be unique. Each problem has a human-written natural-language solution, an annotated count of reasoning steps, and a follow-up question that builds on it, for 2144 items in total. Annotators solved every item themselves to check it.

The authors also propose a scoring scheme that gives more weight to problems needing more steps. This step-weighted score is averaged equally with plain accuracy, and the result is mapped to five reasoning levels. Testing 13 models showed a clear tiering, with GPT-4 variants at the top. Every model scored lower on follow-up questions than on the initial ones. SC-Math6 matters because it gives Chinese LLMs a harder, interaction-aware math testbed whose results are easier to interpret than a single accuracy number.

Key points

  • Dataset design:
    • The dataset has 1072 problem/follow-up pairs (2144 questions).
    • Steps are distributed roughly as follows: 15–20% need 1 step, 15–20% need 2, 45–50% need 3, and 5–10% need 4–5.
    • A spot check found 1 questionable pair out of 50, giving about 98% accuracy.
  • Scoring method:
    • The Comprehensive Score is an equal mix of two parts. The first is a Reasoning Steps Score, weighted by step count. The second is an Overall Accuracy Score, which averages per-question accuracy with strict pair accuracy (both turns must be correct).
    • Models whose scores fall within 5 points of each other share a level.
  • Results:
    • GPT-4-1106-Preview ranked first with 90.71. GPT-4, Ernie-bot 4.0, GLM-4 and Xinghuo 3.5 also reached Level 5.
    • Chinese-Alpaca2-13B scored lowest at 20.55.
  • Follow-up drop:
    • GPT-4-1106-Preview fell from 95.43% to 89.37% on follow-up questions.
    • Xinghuo 3.0 showed the largest drop, losing about 45 points.
  • Other findings:
    • The average model score was 55.34 on SC-Math6, compared with 59.21 on GSM8K.
    • Higher scores were associated with better compliance with output-format instructions and, in some cases, with longer answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksMathematics · 189 worksApplied Mathematics · 52 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Liang Xu, Hang Xue, Lei Zhu and Kangkang Zhao. 2024. SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese. arXiv:2401.11819.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.11819 · arXiv 2401.11819

@misc{xu2024superclue,
  title = {{SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese}},
  author = {Liang Xu, Hang Xue, Lei Zhu and Kangkang Zhao},
  year = {2024},
  doi = {10.48550/arxiv.2401.11819},
  eprint = {2401.11819},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.11819}
}

References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
  2. Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and others, 2021. Training verifiers to solve math word problems. arXiv:2110.14168.
  3. Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob, 2021. Measuring mathematical problem solving with the math dataset. arXiv:2103.03874.
  4. Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others, 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
  5. Li, Xuechen and Zhang, Tianyi and Dubois, Yann and Taori, Rohan and Gulrajani, Ishaan and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B, 2023. Alpacaeval: An automatic evaluator of instruction-following models. GitHub repository. find
  6. Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi, 2019. An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641. find
  7. Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord, 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457. *Figure 5: Relation between reasoning steps and Mean Accuracy*

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2024-superclue-math6-graded-multi/