📄 Over-Reasoning and Redundant Calculation of Large Language Models

Summary

Chain-of-thought (CoT) reasoning helps LLMs on math problems, but it also makes outputs longer, slower and more expensive. This paper asks whether models can tell when step-by-step work is actually needed. The authors build GSM8K-Zero from GSM8K. They drop each question's final query and replace it with a ChatGPT-generated question about a quantity already stated in the text, so no arithmetic is required. GPT-4 then filters the generated pairs. An answer counts as redundant if a regular expression finds the operators ×, + or = in it.

Seven RLHF-trained models were tested zero-shot. Most of them still produced unnecessary calculations, and answers containing calculations were usually less accurate. The authors trace this partly to reward modeling. When GPT-4 and ChatGPT act as proxy reward models, they favour long, calculation-heavy answers even when those answers are wrong.

The study highlights a previously unexamined failure mode: over-reasoning on trivial inputs. It argues for training methods that teach LLMs when step-by-step reasoning is warranted.

Key points

• Dataset construction: GSM8K-Zero was built from 3,500 sampled GSM8K training questions and contains 2,978 question–answer pairs. A manual check of 250 pairs suggests about 85% are valid. It is released on GitHub and Hugging Face (dcml0714/GSM8K-Zero).
• Redundancy rates: Llama-2-chat models (7b, 13b, 70b) included calculations in over 80% of responses. Claude-2 did so in about 75%, ChatGPT in about 47%, PaLM (text-bison-001) in about 29% and GPT-4 in about 12%.
• Accuracy: Several models stayed near or below 50% on this essentially extractive task, including PaLM (~41%) and Llama-2-13b/7b (~40%). For most models, accuracy fell sharply when calculations appeared; for example, ChatGPT dropped from ~97% to ~61%. Typical errors were hallucinated variables and arithmetic mistakes.
• Prompting to skip CoT: Explicitly allowing models to skip CoT reduced redundancy, for instance ChatGPT to ~26% and Llama-2-7b-chat to ~33%, but did not eliminate it.
• Proxy reward models: As proxy reward models, ChatGPT consistently preferred long but wrong answers over short correct ones. GPT-4 chose the short correct answer 61% of the time but still picked the long wrong one in 34% of cases.