📄 Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models

Summary

The paper investigates "overthinking": LLMs, especially reasoning-tuned ones, produce very long chains of thought on trivial arithmetic. These chains often do not improve correctness and can even hurt it. To measure this, the authors built LLMThinkBench. It generates fresh test instances on the fly for 14 deterministic basic-math tasks, including sorting, comparison, arithmetic operations, max/min, mean/median/mode and odd/even counts, which limits contamination. Answers are extracted with a hierarchical parser. The paper also proposes the Overthinking Score, the harmonic mean of accuracy and normalized token efficiency, so a model only scores well if it is both correct and concise. The authors evaluated 53 LLMs, covering standard, reasoning and quantized variants under different reasoning budgets. Reasoning models used far more tokens yet often lost accuracy, and they degraded sharply when output length was capped. Larger reasoning budgets brought little or no benefit. Efficiency-aware rankings favored mid-sized instruction-tuned models. The benchmark is released as a PyPI package with a public leaderboard, which gives a reusable standard for judging reasoning efficiency alongside accuracy.

Key points

• Token use vs. accuracy: reasoning models averaged about 6,780 output tokens versus 378 for standard models, roughly 18× more.
• Collapse under limits: with a 1,024-token cap, accuracy fell from 72% to 44%.
• Reasoning effort: GPT-5, o3 and o4-mini peaked at low effort, and medium or high effort only added tokens without improving accuracy.
• Overthinking Score rankings:
  • GPT-4.1-mini scored 0.930 and GPT-4.1 scored 0.927.
  • Qwen3-14B (0.727) beat Qwen3-32B (0.668).
  • Phi-4-reasoning scored 0.352 against 0.863 for Phi-4.
• Causes of wasted tokens: about 5,000 hand-annotated responses showed four main patterns: redundant verification, self-contradiction, irrelevant exploration and stopping failures. The authors trace these mainly to CoT supervision. Concise prompts and tools (calculator, Python) narrowed the gap but did not close it.
• Quantization: compression barely affected large models (Qwen2.5-32B: 73.08% → 72.67%) but hurt tiny ones badly (Qwen2.5-0.5B: 21.31% → 12.77%).
• Availability: the benchmark installs with pip install llmthinkbench.