Elamur Research⌕
☰
home / papers / 2024 / Do LLMs Really Think Step-by-step In Implicit Reasoning?

Do LLMs Really Think Step-by-step In Implicit Reasoning?

Paper · 2024 · Yijiong Yu

Summary

Implicit chain-of-thought means answering multi-step problems without writing out the intermediate steps. It promises faster, cheaper inference, but it is unclear whether models reason internally the way explicit CoT does. Yu tests this with synthetic addition/subtraction chains of 3–5 steps, where every intermediate value is known.

At every layer, the hidden state of the last prompt token is captured. A separate probe per layer then tries to recover each step's result. Two settings are compared: Qwen2.5-72B-Instruct told to answer directly, and a Mistral-7B model fine-tuned to internalize CoT (Deng et al., 2024).

  • Prompted Qwen: the probes reliably find only the input value and the final answer. Intermediate results are barely detectable, which suggests learned shortcuts rather than genuine stepwise computation.
  • Trained Mistral: it does compute steps one after another across layers. On longer chains, however, it runs out of layers before finishing.
  • Format changes: both models lose much accuracy when equations are reversed or values are divided by 10.

The author concludes that implicit CoT is fragile and that explicit CoT remains necessary.

Key points

  • Data and probing setup:

    • 2000 generated problems, all values kept in the range 0–19.
    • 1600 problems train the probes and 400 test them.
    • Each probe is a 1-layer MLP acting as a 20-class classifier, trained for 8 epochs per layer.

  • Models:

    • Qwen2.5-72B-Instruct, quantized to 4-bit, with its 80 layers averaged pairwise into 40.
    • mistral-7b-internal-CoT, trained on GSM8k, with 32 layers left uncompressed.

  • Layer limit in Mistral: the 2nd step emerges around layer 20 for 3-step problems and around layer 23 for 4-step ones. Steps 4–5 of 5-step problems are never reached.

  • Implicit-CoT accuracy on original / reversed / scaled problems:

    Model3-step5-step
    Mistral78.2 / 13.7 / 10.04.8 / 1.8 / 5.4
    Qwen99.8 / 89.0 / 90.165.2 / 14.9 / 59.1

  • Explicit CoT baseline: Qwen2.5 models at 1.5b, 7b and 72b, prompted to reason step by step, reach close to 100% accuracy regardless of format or step count. The only limitation the author notes is that just arithmetic tasks were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Yijiong Yu. 2024. Do LLMs Really Think Step-by-step In Implicit Reasoning?. arXiv:2411.15862.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.15862 · arXiv 2411.15862

@misc{yu2024do,
  title = {{Do LLMs Really Think Step-by-step In Implicit Reasoning?}},
  author = {Yijiong Yu},
  year = {2024},
  doi = {10.48550/arxiv.2411.15862},
  eprint = {2411.15862},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.15862}
}

References

11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. Preprint, arxiv:2309.14402 [cs].
  2. Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From explicit CoT to implicit CoT: Learning to internalize CoT step by step. Preprint, arxiv:2405.14838 [cs]. find
  3. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowledge distillation. Preprint, arxiv:2311.01460. find
  4. Daniel Kahneman. 2011. Thinking, fast and slow. Farrar, Straus and Giroux. find
  5. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 2024. O1 Replication Journey: A Strategic Progress Report – Part 1. arXiv preprint. ArXiv:2410.18982.
  6. Qwen Team. 2024. Qwen2.5: A party of foundation models! Section: blog. find
  7. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. Preprint, arxiv:2405.15071. find
  8. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi…
  9. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? Preprint, arxiv:2402.16837. find
  10. Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2024. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process.
  11. Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023. Towards Better Chain-of-Thought Prompting Strategies: A Survey. arXiv preprint. ArXiv:2310.04959. — [arXiv:2310.04959](https://arxiv.org/abs/2310.04959) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2024-llms-really-think-step-step/