Elamur Research⌕
☰
home / papers / 2026 / Think Like You Execute: Verifiable Chain of Thought from Program Traces

Think Like You Execute: Verifiable Chain of Thought from Program Traces

Paper · 2026 · Shailja Thakur, Vaibhav Saxena, Rohan Kulkarni et al.

Summary

Synthetic chain-of-thought data for code is usually written by teacher LLMs. Its explanations can sound convincing while misreporting variable values or control flow, and models trained on it learn those errors. The authors (IBM Research) built an open-source Python pipeline that grounds every rationale in actual execution.

The pipeline has three stages:

  • Concepts and problems. Programming concepts are mined from permissive sources such as StarCoder2-documentation and turned into problems, solutions and unit tests.
  • Code-test selection. An adapted Dual Agreement scheme keeps the solution cluster that passes the most shared tests.
  • Trace-based rationales. Solutions are instrumented with pysnooper, and an LLM narrates the cleaned traces as forward (input→output) and backward (output→input) reasoning. Each rationale sentence is checked against the trace with a sliding window, and the predicted I/O is compared with the test case. Rationales that fail are dropped.

The result is 54,000 verified bi-directional samples. Fine-tuned models improved by up to +26.6 on LiveCodeBench-Exec, +22.2 on CruxEval and +19.5 on HumanEval. This indicates that verified data improves both code reasoning and code generation.

Key points

  • Stage A yields roughly 8,000 seed concepts. Trace sanitization (removing ANSI codes, timestamps and file paths) shortens traces by about 40% on average.

  • In the 25k-problem ablation, LiveCodeBench scores were:

    • base model: 46.3
    • LLM-judged CoT without execution grounding: 59.3
    • minimal execution (1 solution, 1 test): 62.1
    • full approach: 68.2

    HumanEval Pass@1 rose from 62.0 to 81.5.

  • Fine-tuning instruct models also helped: +39.9 on CruxEval Input for Granite-3.3-8B-Instruct and +21.5 on CruxEval Output for Qwen2.5-Coder-7B-Instruct.

  • CoTs from the fine-tuned Qwen2.5-Coder-7B were 761% richer in technical-token entropy than the base model's. Their reasoning was also more consistent with the final answer (R² 0.122 vs 0.011).

  • The 25k verified subset was used in post-training of Granite-family models, and the pipeline has been adapted to OpenInstruct. Only Python is supported for now; Java is planned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬IBM Research - IndiaNew Delhi · India9 works · 749 citations🏢IBM (India)Bengaluru · India6 works · 919 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksSoftware · 92 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Shailja Thakur, Vaibhav Saxena, Rohan Kulkarni, Shivdeep Singh, Parameswaran Selvam, Hiroshi Kanayama and Hima Patel. 2026. Think Like You Execute: Verifiable Chain of Thought from Program Traces. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.53

@inproceedings{thakur2026think,
  title = {{Think Like You Execute: Verifiable Chain of Thought from Program Traces}},
  author = {Shailja Thakur, Vaibhav Saxena, Rohan Kulkarni, Shivdeep Singh, Parameswaran Selvam, Hiroshi Kanayama and Hima Patel},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.53},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.53}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jordi Armengol-Estapé, Quentin Carbonneaux, Tianjun Zhang, Aram H. Markosyan, Volker Seeker, Chris Cummins, Melanie Kambadur, Michael F. P. O’Boyle, Sida Wang, Gabriel Synnaeve, and Hugh James Leather. 2025. What i cannot execute, i do not understand: Training and evaluating llms on program executio… — [arXiv:2503.05703](https://arxiv.org/abs/2503.05703) find
  2. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. Preprint, arXiv:2401.14196. find
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  4. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. Qwen2.5-coder technical report. Preprint, arXiv:2409.12186.
  5. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. Preprint, arXiv:2207.10397. find
  6. Justin Chih-Yao Chen, Zifeng Wang, Hamid Palangi, Rujun Han, Sayna Ebrahimi, Long Le, Vincent Perot, Swaroop Mishra, Mohit Bansal, Chen-Yu Lee, and Tomas Pfister. 2025. Reverse thinking makes llms stronger reasoners. Preprint, arXiv:2411.19865. find
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023a. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Preprint, arXiv:2211.12588. find
  9. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023b. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. find
  10. Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. Semcoder: Training code language models with comprehensive semantics reasoning. Preprint, arXiv:2406.01006. — [arXiv:2406.01006](https://arxiv.org/abs/2406.01006) find
  11. Yangruibo Ding, Ben Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2023. Traced: Execution-aware pre-training for source code. Preprint, arXiv:2306.07487. — [arXiv:2306.07487](https://arxiv.org/abs/2306.07487) find
  12. EbookFoundation. 2024. Free programming books. https://github.com/EbookFoundation/free-pro gramming-books. ↗
  13. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. Preprint, arXiv:2401.03065.
  14. IBM Research. 2025. IBM Granite 4.0: Hyper-Efficient High-Performance Hybrid Models. https://www.ibm.com/new/announcements/ibm-granite-4 -0-hyper-efficient-high-performance-hybri d-models. ↗
  15. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. Preprint, arXiv:2403.07974.
  16. Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T. Kwok. 2024. Forward-backward reasoning in large language models for mathematical verification. Preprint, arXiv:2308.07758. find
  17. Dongwon Jung, Wenxuan Zhou, and Muhao Chen. 2025. Code execution as grounded supervision for llm reasoning. Preprint, arXiv:2506.10343. find
  18. Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. 2025. Codei/o: Condensing reasoning patterns via code input-output prediction. Preprint, arXiv:2502.07316.
  19. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, and 1 others. 2023. Starcoder: may the source be with you! Preprint, arXiv:2305.06161. find
  20. Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, and Mao Yang. 2025. rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset. Preprint, arXiv:2505.21297.
  21. Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan-Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Raju Pavuluri, and 27 oth… find
  22. Mistral AI. 2025. Mistral small 3. https://mistral.ai/news/mistral-small-3/. ↗
  23. Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404.14662. find
  24. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21…
  25. Team OLMo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, and Others. 2025. Olmo 3. Preprint, arXiv:2512.13961.
  26. Ram Rachum, Alex Hall, Iori Yanokura, and 1 others. 2019. Pysnooper: Never use print for debugging again. find
  27. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, a… — Code Llama: Open Foundation Models for Code
  28. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366.
  29. StarCoder2-Documentation. 2024. Starcoder 2 and the stack v2: The next generation. https://huggingf ace.co/datasets/SivilTaram/starcoder2-doc umentation. find
  30. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Preprint, arXiv:2305.04388.
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023a. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  32. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023b. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thakur-2026-think-like-you-execute-verifiable/