Elamur Research⌕
☰
home / papers / 2026 / LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

Paper · 2026 · Lihao Sun, Hang Dong, Bo Qiao et al.

Summary

The paper asks whether chain-of-thought reasoning in an LLM follows an organized path through hidden-state space, and whether that path can be used to diagnose and steer reasoning. The authors prompt three Llama 3.1 8B variants (Base, Instruct, R1-Distill) to solve GSM8K and MATH-500 problems with explicit "Step" markers. They record activations at the token just before each marker and before the final-answer marker. Linear probes and t-SNE show that each step index occupies its own region, and these regions separate more cleanly in deeper layers. This organization already exists in the Base model. Reasoning distillation mainly causes earlier convergence toward a region linked to finishing the reasoning. Correct and incorrect solutions look alike in early steps but drift apart in later ones, which lets the authors predict answer correctness before the answer appears. Building on this, they propose trajectory-based steering, which can repair failing reasoning or lengthen or shorten it. The work gives a geometric tool for interpreting, monitoring and controlling LLM reasoning.

Key points

  • Data and models: GSM8K (7,473 train / 1,319 test) and MATH-500 (500 questions), with Llama 3.1 8B Base, Instruct and R1-Distill under deterministic decoding. Code is released at github.com/slhleosun/reasoning-trajectory.
  • Step separability: Probes identify Step 1 with accuracy above 0.99 at every layer. For Step 5, accuracy at layer 0 is only 0.58–0.77 and passes 0.90 only after layer 6. Step probes largely transfer across the three models.
  • Correctness prediction: Linear classifiers on late-step features reach ROC–AUC up to 0.87 before the final answer is emitted. Using this prediction to gate interventions gives small but consistent accuracy gains.
  • Trajectory-based steering: Activations that stray from a PCA-based "ideal" path of correct solutions receive low-rank corrections. Accuracy rises from 75.44% to 83.04% on six-step GSM8K problems and from 67.69% to 75.38% on seven-step problems, while at least 97% of already-correct answers are preserved. There is almost no effect on problems with five or fewer steps.
  • Length control: Steering toward or away from the termination subspace shortens or extends reasoning roughly monotonically for |α| ≤ 0.8. Accuracy changes by about 1% when |α| ≤ 0.4. At larger strengths the model falls into repetitive loops.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Lihao Sun, Hang Dong, Bo Qiao, Qingwei Lin, Dongmei Zhang and Saravan Rajmohan. 2026. LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1237

@inproceedings{sun2026llm,
  title = {{LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals}},
  author = {Lihao Sun, Hang Dong, Bo Qiao, Qingwei Lin, Dongmei Zhang and Saravan Rajmohan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1237},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1237}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. Preprint, arXiv:2503.08679. \#\# Ethical Considerations This work analyzes internal representations and inference-time interven… find
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  3. Deepseek. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  4. Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. 2024. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. Preprint, arXiv:2402.18312. find
  5. Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. 2025. Does thinking more always help? mirage of test-time scaling in reasoning models. Preprint, arXiv:2506.04210. — [arXiv:2506.04210](https://arxiv.org/abs/2506.04210) find
  6. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  7. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, an… — Measuring Faithfulness in Chain-of-Thought Reasoning
  8. Andrew Lee, Lihao Sun, Chris Wendler, Fernanda Viégas, and Martin Wattenberg. 2025. The geometry of self-verification in a task-specific reasoning model. Preprint, arXiv:2504.14379. — [arXiv:2504.14379](https://arxiv.org/abs/2504.14379) find
  9. Bo Li, Guanzhi Deng, Ronghao Chen, Junrong Yue, Shuo Zhang, Qinghua Zhao, Linqi Song, and Lijie Wen. 2025. Rema: A unified reasoning manifold framework for interpreting large language model. Preprint, arXiv:2509.22518. — [arXiv:2509.22518](https://arxiv.org/abs/2509.22518) find
  10. Zhenwen Liang, Ruosen Li, Yujun Zhou, Linfeng Song, Dian Yu, Xinya Du, Haitao Mi, and Dong Yu. 2025. Clue: Non-parametric verification from experience via hidden-state clustering. Preprint, arXiv:2510.01591. — [arXiv:2510.01591](https://arxiv.org/abs/2510.01591) find
  11. Jiarui Liu, Jivitesh Jain, Mona Diab, and Nishant Subramani. 2025a. Llm microscope: What model internals reveal about answer correctness and context utilization. Preprint, arXiv:2510.04013. — [arXiv:2510.04013](https://arxiv.org/abs/2510.04013) find
  12. Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. 2025b. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. Preprint, arXiv:2410.21333. — [arXiv:2410.21333](https://arxiv.org/abs/2410.21333) find
  13. Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  14. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Preprint, arXiv:2501.19393.
  15. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39643–39666. PMLR.
  16. Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. 2025. Demystifying reasoning dynamics with mutual information: Thinking tokens are information peaks in llm reasoning. Preprint, arXiv:2506.02867. — [arXiv:2506.02867](https://arxiv.org/abs/2506.02867) find
  17. Lihao Sun, Lewen Yan, Xiaoya Lu, Andrew Lee, Jie Zhang, and Jing Shao. 2026. Valence-arousal subspace in llms: Circular emotion geometry and multi-behavioral control. Preprint, arXiv:2604.03147. — [arXiv:2604.03147](https://arxiv.org/abs/2604.03147) find
  18. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  19. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, pages 74952– 74965.
  20. Curran Associates, Inc. 2025. Wait, we don’t need to "wait"! removing thinking tokens improves reasoning efficiency. Preprint, arXiv:2506.08343. find
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  22. Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. 2025a. Reasoning models know when they’re right: Probing hidden states for self-verification. Preprint, arXiv:2504.05419. find
  23. Fred Zhang and Neel Nanda. 2024. Towards best practices of activation patching in language models: Metrics and methods. Preprint, arXiv:2309.16042. find
  24. Junyu Zhang, Runpei Dong, Han Wang, Xuying Ning, Haoran Geng, Peihao Li, Xialin He, Yutong Bai, Jitendra Malik, Saurabh Gupta, and Huan Zhang. 2025b. Alphaone: Reasoning models thinking slow and fast at test time. Preprint, arXiv:2505.24863.
  25. James Xu Zhao, Bryan Hooi, and See-Kiong Ng. 2025. Test-time scaling in reasoning models is not effective for knowledge-intensive tasks yet. Preprint, arXiv:2509.06861. — [arXiv:2509.06861](https://arxiv.org/abs/2509.06861) find
  26. Yue Zhou, Yada Zhu, Diego Antognini, Yoon Kim, and Yang Zhang. 2024. Paraphrase and solve: Exploring and exploiting the impact of surface form on mathematical reasoning in large language models. Preprint, arXiv:2404.11500. find
  27. Yufa Zhou, Yixiao Wang, Xunjian Yin, Shuyan Zhou, and Anru R. Zhang. 2025a. The geometry of reasoning: Flowing logics in representation space. Preprint, arXiv:2510.09782. — [arXiv:2510.09782](https://arxiv.org/abs/2510.09782) find
  28. Zhanke Zhou, Zhaocheng Zhu, Xuan Li, Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, and Bo Han. 2025b. Landscape of thoughts: Visualizing the reasoning process of large language models. Preprint, arXiv:2503.22165. — [arXiv:2503.22165](https://arxiv.org/abs/2503.22165) find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  30. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. Towards understanding chain-of-thought prompting: An empirical study of what matters. Preprint, arXiv:2212.10001.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-llm-reasoning-as-trajectories-step/