Elamur Research⌕
☰
home / papers / 2026 / Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

Paper · 2026 · Chenming Tang, Hsiu-Yuan Huang, Weijie Liu et al.

Summary

RLVR training such as GRPO learns only from the model's own sampled answers. When every rollout for a hard problem is wrong, the group-relative advantages all become zero and the problem yields no gradient. The authors call this exploration stagnation. Existing fixes depend on human-written solutions or traces from stronger models, which are costly or sometimes unavailable.

The proposed method, LTE, gets around that dependency by having the model pick up reasoning through its own trial and error and reuse its failures. For a "none-pass" problem, LTE runs a second batch of rollouts:

  • Wrong answers as hints: final answers extracted from the failed attempts go into the prompt as known-incorrect options, which narrows the search space.
  • Concise-thinking hint: if some or all responses were cut off for length, the prompt also asks for concise reasoning.

Correct hinted rollouts are added back to the group and trained with mixed-policy GRPO using regularized importance sampling.

On six math benchmarks with Qwen3-8B-Base, LTE beats standard GRPO and even LUFFY, which uses external solutions. LTE also shrinks the pool of unsolved training problems. This matters because LTE is self-sufficient guidance that scales without stronger teacher models.

Key points

  • With Qwen3-8B-Base, LTE improves over GRPO by 5.02 Pass@1 and 9.96 Pass@k, averaged over MATH-500, Minerva, OlympiadBench, AMC'23, AIME'24 and AIME'25.
  • Compared with plain extra rollouts without hints (GRPO_Extra), the averages rise from 41.72/56.74 to 49.01/66.78 (Pass@1/Pass@k).
  • In the ablation, the incorrect-answer hint matters more than the concise hint: removing it gives 44.47/62.48, while removing the concise hint gives 47.19/65.07.
  • The variant with entropy loss, LTE†, outperforms LUFFY by 2.41 Pass@1 and 2.15 Pass@k.
  • On SciQ with Qwen3-4B-Base, LTE reaches 57.75/59.83 on the test set (Pass@1/Pass@4), versus 55.43/57.35 for GRPO.
  • Stated limitations:
    • Training used a maximum response length of 8,192 tokens.
    • Larger models were not tested.
    • The method targets short-answer domains such as math and science, not coding or agent tasks.
  • Code is released at github.com/JamyDon/LTE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢TencentShenzhen · China134 works · 19,743 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Clive Bai, Saiyong Yang and Yunfang Wu. 2026. Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.363

@inproceedings{tang2026do,
  title = {{Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error}},
  author = {Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Clive Bai, Saiyong Yang and Yunfang Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.363},
  url = {https://doi.org/10.18653/v1/2026.acl-long.363}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa Alazraki, Maximilian Mozes, Jon Ander Campos, Tan Yi-Chern, Marek Rei, and Max Bartolo. 2025. No need for explanations: LLMs can implicitly learn from mistakes in-context. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. find
  2. Justin Chih-Yao Chen, Becky Xiangyu Peng, Prafulla Kumar Choubey, Kung-Hsiang Huang, Jiaxin Zhang, Mohit Bansal, and Chien-Sheng Wu. 2025a. Nudging the boundaries of llm reasoning. Preprint, arXiv:2509.25666. — [arXiv:2509.25666](https://arxiv.org/abs/2509.25666) find
  3. Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. 2025b. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models. Preprint, arXiv:2508.10751. — [arXiv:2508.10751](https://arxiv.org/abs/2508.10751) find
  4. Stephen Chung, Wenyu Du, and Jie Fu. 2025. Learning from failures in multi-attempt reinforcement learning. Preprint, arXiv:2503.04808. — [arXiv:2503.04808](https://arxiv.org/abs/2503.04808) find
  5. Use of AI Assistants The algorithmic design and core methodology of this work were derived through manual research and human reasoning. For coding, we work with the assistance of GitHub Copilot 3 . We certify that the substance of our code is our own work. https://github.com/features/copilot Reprodu… ↗
  6. Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. 2025. Online difficulty filtering for reasoning oriented reinforcement learning. Preprint, arXiv:2504.03380. *Table 7: Licenses of scientific artifacts we use.* Computational Budget All our experiments are conduc…
  7. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. 2025. The entropy mechanism of reinforcement learning for reasoning language models. Preprint, a… find
  8. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  9. Shihan Dou, Muling Wu, Jingwen Xu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. Improving rl exploration for llm reasoning through retrospective replay. Preprint, arXiv:2504.14363. — [arXiv:2504.14363](https://arxiv.org/abs/2504.14363) find
  10. Zeyu Gan, Hao Yi, and Yong Liu. 2025. Cot-space: A theoretical framework for internal slow-thinking via reinforcement learning. Preprint, arXiv:2509.04027. — [arXiv:2509.04027](https://arxiv.org/abs/2509.04027) find
  11. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proce…
  12. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. 2025. Skywork open reasoner 1 technical report. Preprint, arXiv:2505.22312. find
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  14. Jian Hu, Mingjie Liu, Ximing Lu, Fang Wu, Zaid Harchaoui, Shizhe Diao, Yejin Choi, Pavlo Molchanov, Jun Yang, Jan Kautz, and Yi Dong. 2025. Brorl: Scaling reinforcement learning via broadened exploration. Preprint, arXiv:2510.01180. — [arXiv:2510.01180](https://arxiv.org/abs/2510.01180) find
  15. Kimi Team. 2025. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534.
  16. Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. I…
  17. Ziniu Li, Congliang Chen, Tianyun Yang, Tian Ding, Ruoyu Sun, Ge Zhang, Wenhao Huang, and Zhi-Quan Luo. 2025. Knapsack rl: Unlocking exploration of llms via optimizing budget allocation. Preprint, arXiv:2509.25849. — [arXiv:2509.25849](https://arxiv.org/abs/2509.25849) find
  18. Huanyu Liu, Jia Li, Chang Yu, Taozhi Chen, Yihong Dong, Lecheng Wang, XiaoLong Hu, and Ge Li. 2025a. Evocot: Overcoming the exploration bottleneck in reinforcement learning. Preprint, arXiv:2508.07809. — [arXiv:2508.07809](https://arxiv.org/abs/2508.07809) find
  19. Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li. 2025b. A simple "try again" can elicit multi-turn llm reasoning. Preprint, arXiv:2507.14295. find
  20. Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  21. Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, Ning Ding, and Bowen Zhou. 2025. Towards a unified view of large language model post-training. Preprint, arXiv:2509.04419. — [arXiv:2509.04419](https://arxiv.org/abs/2509.04419) find
  22. Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, and Wentao Zhang. 2025. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. Preprint, arXiv:2506.07527. — [arXiv:2506.07527](https://arxiv.org/abs/2506.07527) find
  23. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning. find
  24. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  25. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems.
  26. Yuda Song, Julia Kempe, and Remi Munos. 2025. Outcome-based exploration for llm reasoning. Preprint, arXiv:2509.06941. find
  27. Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri. 2024. An in-context learning agent for formal theorem-proving. In First Conference on Language Modeling. find
  28. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. Preprint, arXiv:1707.06209.
  29. Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, Yanwei Fu, Qin Liu, Songyang Zhang, and Qi Zhang. 2025. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. Preprint, arXiv:2507.1053… — [arXiv:2507.10532](https://arxiv.org/abs/2507.10532) find
  30. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. Preprint, arXiv:2504.14945. find
  31. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  32. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:24… find
  33. Wenkai Yang, Weijie Liu, Ruobing Xie, Yiju Guo, Lulu Wu, Saiyong Yang, and Yankai Lin. 2025c. Laser: Reinforcement learning with last-token self-rewarding. Preprint, arXiv:2510.14943. — [arXiv:2510.14943](https://arxiv.org/abs/2510.14943) find
  34. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. Dapo: An open-source llm reinforcement learning s…
  35. Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. 2025a. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. Preprint, arXiv:2507.02841.
  36. Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. 2025b. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. Preprint, arXiv:2508.11408. — [arXiv:2508.11408](https://arxiv.org/abs/2508.11408) find
  37. Yuheng Zhang, Wenlin Yao, Changlong Yu, Yao Liu, Qingyu Yin, Bing Yin, Hyokun Yun, and Lihong Li. 2025c. Improving sampling efficiency in rlvr through adaptive rollout and response reuse. Preprint, arXiv:2509.25808. — [arXiv:2509.25808](https://arxiv.org/abs/2509.25808) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2026-do-not-step-into-same/