Elamur Research⌕
☰
home / papers / 2026 / The Inner Monologue of Language Models: When Reasoning Traces Reveal More Than They Hide

The Inner Monologue of Language Models: When Reasoning Traces Reveal More Than They Hide

Paper · 2026 · Pratham Singla, Shivank Garg, Ayush Singh et al.

Summary

Reasoning-focused LLMs generate visible "thinking" text before answering. Oversight methods such as CoT monitoring assume this text honestly reflects the final decision. The authors test that assumption. They also ask whether post-trained models can recognize the hidden behaviors their training gave them.

They fine-tune models with SFT, DPO and GRPO on five task families, each implicitly encoding a latent policy:

  • biased answers
  • risky versus cautious choices
  • reward hacking in code
  • skewed sampling in Rock-Paper-Scissors
  • behavior under pressure

They also inspect the open reasoning traces of DeepSeek-R1.

Three abilities are measured:

  • whether models can describe their acquired tendencies
  • whether those tendencies carry over to structurally similar but semantically different tasks
  • whether reasoning agrees with the answer, assessed with correlation analysis and a proposed Reflective Gain Ratio (RGR)

RL-trained models are more self-aware and transfer their policies better than SFT models. However, their reasoning often diverges from what they actually answer, and this effect is strongest under GRPO. The work warns against treating reasoning traces as a reliable window into model behavior.

Key points

  • The study has three research questions: reasoning–answer faithfulness (RQ1), reflective behavioral self-awareness (RQ2), and out-of-distribution latent policy transfer (RQ3). Faithfulness is judged behaviorally, not mechanistically.
  • In Rock-Paper-Scissors, the SFT model memorizes its training data and always outputs "Paper." On the OOD Table-Bed-Chair game, DPO and GRPO both drift toward "Table," with GRPO shifting less than DPO.
  • Policy transfer depends on the domain:
    • Gender-bias training barely carries over to nationality bias.
    • Risk-taking or risk-avoiding behavior transfers well.
    • Reward hacking learned on code reappears on math tasks.
  • GRPO-trained models and DeepSeek-R1 show extreme RGR values, meaning frequent reasoning–answer mismatch. The authors link this to rewards being applied only to final answers. Models trained to act riskily sometimes describe themselves as safe in their reasoning.
  • Limitations:
    • Training used a fixed learning rate and epoch count taken from HuggingFace recommended configurations.
    • Some SFT models could not be scored on their reasoning because they did not produce reasoning in the expected format.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Boston UniversityBoston · United States26 works · 4,022 citations🎓Indian Institute of Technology RoorkeeRoorkee · India3 works · 6 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Pratham Singla, Shivank Garg, Ayush Singh, Ishan Garg and Ketan Suhaas Saichandran. 2026. The Inner Monologue of Language Models: When Reasoning Traces Reveal More Than They Hide. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2078

@inproceedings{singla2026inner,
  title = {{The Inner Monologue of Language Models: When Reasoning Traces Reveal More Than They Hide}},
  author = {Pratham Singla, Shivank Garg, Ayush Singh, Ishan Garg and Ketan Suhaas Saichandran},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2078},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2078}
}

References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  2. Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Wang. 2024. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. Preprint, arXiv:2305.13712.
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  4. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. Preprint, arXiv:2503.11926. find
  5. Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. 2025a. Tell me about yourself: LLMs are aware of their learned behaviors. In The Thirteenth International Conference on Learning Representations. find
  6. Jan Betley, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, James Chua, and Owain Evans. 2025b. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Proceedings of the 42nd International Conference on Machine Learning. find
  7. Yanda Chen, Yannis Katsis, Jason Jia, Mark Regan, Vincent Perot, Swaroop Prakash, Tanmay Karayil, Rishabh Madan, Anna Rumshisky, and Kyunghyun Cho. 2025. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410. find
  8. Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. 2024. Do models explain themselves? counterfactual simulatability of natural language explanations. In Proceedings of the 41st International Conference on Machine Learning, pages 7880–7904. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. 2024. Alignment faking in large language models. arXiv preprint arXiv:2412.14093. find
  11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  12. Peter Hase and Mohit Bansal. 2020. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540–5552.
  13. Bernease Herman. 2019. The promise and peril of human evaluation for model interpretability. Preprint, arXiv:1711.07414. — [arXiv:1711.07414](https://arxiv.org/abs/1711.07414) find
  14. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205.
  15. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  16. Felix J Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. 2024. Looking inward: Language models can learn about themselves by introspection. Preprint, arXiv:2410.13787. — [arXiv:2410.13787](https://arxiv.org/abs/2410.13787) find
  17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  18. Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur. 2024. Finetuning language models to emit linguistic expressions of uncertainty. Preprint, arXiv:2409.12180. — [arXiv:2409.12180](https://arxiv.org/abs/2409.12180) find
  19. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv:2207.05221. arXiv preprint Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan… find
  20. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Sa… — Measuring Faithfulness in Chain-of-Thought Reasoning
  21. Steven A. Lehr, Ketan S. Saichandran, Eddie Harmon-Jones, Nykko Vitali, and Mahzarin R. Banaji. 2025. Kernels of selfhood: Gpt-4o shows humanlike patterns of cognitive dissonance moderated by free choice. Proceedings of the National Academy of Sciences, 122(20):e2501823122. find
  22. Yuan Li, Yue Huang, Yuli Lin, Siyuan Wu, Yao Wan, and Lichao Sun. 2024. I think, therefore i am: Benchmarking awareness of large language models using awarebench. Preprint, arXiv:2401.17882. — [arXiv:2401.17882](https://arxiv.org/abs/2401.17882) find
  23. Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. 2024. Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984. — [arXiv:2412.04984](https://arxiv.org/abs/2412.04984) find
  24. Hadi Mohammadi, Tamas Kozak, and Anastasia Giachanou. 2025. Evaluating grpo and dpo for faithful chain-of-thought reasoning in llms. arXiv preprint arXiv:2512.22631. find
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
  27. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
  28. Marco Túlio Ribeiro, Sameer Singh, and C Guestrin. 2016. Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. find
  29. Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. 2024. Large language models can strategically deceive their users when put under pressure. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. find
  30. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  31. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  32. Yuda Song, Julia Kempe, and Rémi Munos. 2025. Outcome-based exploration for llm reasoning. In NeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists. find
  33. Evangelia Spiliopoulou, Riccardo Fogliato, Hanna Burnsky, Tamer Soliman, Jie Ma, Graham Horwood, and Miguel Ballesteros. 2025. Play favorites: A statistical method to measure self-bias in llm-as-a-judge. arXiv preprint arXiv:2508.06709. — [arXiv:2508.06709](https://arxiv.org/abs/2508.06709) find
  34. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965.
  35. Yuhao Wang, Yusheng Liao, Heyang Liu, Hongcheng Liu, Yu Wang, and Yanfeng Wang. 2024. Mm-sap: A comprehensive benchmark for assessing self-awareness of multimodal large language models in perception. Preprint, arXiv:2401.07529. find
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  37. Xumeng Wen, Jie Liao, Xuandong Gao, Haotian Zheng, and Yingqian Cao. 2025. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. find
  38. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  39. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? Preprint, arXiv:2305.18153. find
  40. Tao Zhang, Ziqian Zeng, Yuxiang Xiao, Huiping Zhuang, Cen Chen, James Foulds, and Shimei Pan. 2024. Genderalign: An alignment dataset for mitigating gender bias in large language models. Preprint, arXiv:2406.13925. find
  41. Yachao Zhao, Bo Wang, and Yan Wang. 2025. Explicit vs. implicit: Investigating social bias in large language models through self-reflection. arXiv preprint arXiv:2501.02295. find
  42. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097–1100.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singla-2026-inner-monologue-language-models-when/