Elamur Research⌕
☰
home / papers / 2026 / Self-Reflective Generation at Test Time

Self-Reflective Generation at Test Time

Paper · 2026 · Jian Mu, Qixin Zhang, Zhiyong Wang et al.

Summary

Autoregressive LLMs cannot revise a token once it is emitted, so one early mistake in a long chain-of-thought can derail the whole solution. Existing fixes either rewrite complete drafts afterwards or teach self-correction through costly training such as RL. Both act only after the error already exists. SRGen is a training-free, test-time alternative that intervenes before a risky token is committed. During decoding, it tracks next-token entropy and compares it with a threshold built from the mean and standard deviation of recent entropies. When that threshold is crossed, it briefly optimizes a small vector δ and adds it to the final hidden state. The optimization objective balances two goals: keeping the already generated prefix probable and sharpening the current prediction. Experiments on math reasoning, general reasoning and code across several LLMs show consistent accuracy gains with modest extra latency. The method also combines well with techniques such as SLOT. Its value lies in being a cheap plug-in for more reliable reasoning, although it requires white-box access to hidden states and gradients.

Key points

  • Trigger rule. Reflection starts when the current entropy exceeds μ + k·σ, computed over a sliding window of the last N token entropies. This makes the trigger adapt to each model's entropy profile.
  • Loss. δ is trained with a hybrid loss L_SRGen = (1−λ)·L_CE + λ·L_AEM.
    • L_CE is a retrospective cross-entropy over the prefix, with δ applied to all past hidden states.
    • L_AEM is an entropy-minimization term.
  • Combination with SLOT on Qwen2.5-Math-7B with greedy decoding:
    • MATH500: 63.8% → 70.6%
    • AMC: 35.0% → 38.0% (SRGen alone reaches 37.0%)
    • AIME2024: 13.3% → 30.0%
  • What gets flagged. In DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B on MATH500 and GPQA, flagged tokens are mostly function words and connectives (e.g., "so", "but", "maybe", "if"). These tend to sit at reasoning decision points.
  • Availability and limitations. Code is released at github.com/2020-qqtcg/SRGen. Limitations include the need for white-box inference, extra hyperparameters, and position selection that could still be improved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China56 works · 3,212 citations🎓City University of Hong KongHong Kong · China59 works · 3,480 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China67 works · 1,765 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Jian Mu, Qixin Zhang, Zhiyong Wang, Menglin Yang, Shuang Qiu, Chengwei Qin, Zhongxiang Dai and Yao Shu. 2026. Self-Reflective Generation at Test Time. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.465

@inproceedings{mu2026self,
  title = {{Self-Reflective Generation at Test Time}},
  author = {Jian Mu, Qixin Zhang, Zhiyong Wang, Menglin Yang, Shuang Qiu, Chengwei Qin, Zhongxiang Dai and Yao Shu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.465},
  url = {https://doi.org/10.18653/v1/2026.acl-long.465}
}

References

41 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Art of Problem Solving. 2024. Aime problems and solutions. https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_ Solutions. Accessed: 2025-09-14. ↗
  2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  3. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. Matharena: Evaluating llms on uncontaminated math competitions.
  4. Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mozolevskyi, Muayad Ali, and Waseem AlShikh. 2025. Reflect, retry, reward: Self-improving llms via reinforcement learning. arXiv preprint arXiv:2505.24726. — [arXiv:2505.24726](https://arxiv.org/abs/2505.24726) find
  5. Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. 2024. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078.
  6. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations.
  7. Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. arXiv preprint arXiv:2508.15260.
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  9. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  10. Hasan Abed Al Kader Hammoud, Hani Itani, and Bernard Ghanem. 2025. Beyond the last answer: Your reasoning trace uncovers more than you think. arXiv preprint arXiv:2504.20708. — [arXiv:2504.20708](https://arxiv.org/abs/2504.20708) find
  11. Viktor Moskvoretskii, Chris Biemann, and Irina Nikishina. 2025. Self-taught self-correction for small language models. arXiv preprint arXiv:2503.08681. find
  12. Chi Hu, Yimin Hu, Hang Cao, Tong Xiao, and Jingbo Zhu. 2024. Teaching language models to self-improve by learning from language feedback. arXiv preprint arXiv:2406.07168. find
  13. Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. arXiv preprint arXiv:2402.13950.
  14. Yang Hu, Xingyu Zhang, Xueji Fang, Zhiyang Chen, Xiao Wang, Huatian Zhang, and Guojun Qi. 2025. Slot: Sample-specific language model optimization at test-time. arXiv preprint arXiv:2505.12392. — [arXiv:2505.12392](https://arxiv.org/abs/2505.12392) find
  15. Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. 2024. Reasoning with large language models, a survey. CoRR.
  16. Kushal Jain, Moritz Miller, Niket Tandon, and Kumar Shridhar. 2025. First-step advantage: Importance of starting right in multi-step math reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 766–778, Vienna, Austria. Association for Computational Linguistics. find
  17. Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. 2025. Demystifying reasoning dynamics with mutual information: Thinking tokens are information peaks in llm reasoning. arXiv preprint arXiv:2506.02867. — [arXiv:2506.02867](https://arxiv.org/abs/2506.02867) find
  18. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, and 1 others. 2024. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917.
  19. Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. find
  20. Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, and Yu Cheng. 2025. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. arXiv preprint arXiv:2501.12895. — [arXiv:2501.12895](https://arxiv.org/abs/2501.12895) find
  21. Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. 2025. Critical tokens matter: Token-level contrastive estimation enhances llm’s reasoning capability. In Forty-second International Conference on Machine Learning. find
  22. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572.
  23. Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. 2025. S 2 r: Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502.12853.
  24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  25. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  26. Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. 2025. When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning. arXiv preprint arXiv:2504.01005.
  27. Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. 2025. Confidence improves self-consistency in llms. arXiv preprint arXiv:2502.06233. find
  28. Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  29. Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. 2025. Atom of thoughts for markov llm test-time scaling. arXiv preprint arXiv:2502.12018. — [arXiv:2502.12018](https://arxiv.org/abs/2502.12018) find
  30. Jean Vassoyan, Nathanaël Beau, and Roman Plaud. 2025. Ignore the kl penalty! boosting exploration on critical tokens to enhance rl fine-tuning. arXiv preprint arXiv:2502.06533. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  32. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and 1 others. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939.
  33. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  36. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  37. Xiao Yu, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhou Yu. 2024. Teaching language models to self-improve through interactive demonstrations. In NAACL-HLT. find
  38. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496.
  39. Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, and 1 others. 2025. First return, entropy-eliciting explore. arXiv preprint arXiv:2507.07017. — [arXiv:2507.07017](https://arxiv.org/abs/2507.07017) find
  40. Zhi Zhou, Tan Yuhao, Zenan Li, Yuan Yao, Lan-Zhe Guo, Xiaoxing Ma, and Yu-Feng Li. 2025. Bridging internal probability and self-consistency for effective and efficient llm reasoning. arXiv preprint arXiv:2502.00511. — [arXiv:2502.00511](https://arxiv.org/abs/2502.00511) find
  41. Yuqi Zhu, Ge Li, Xue Jiang, Jia Li, Hong Mei, Zhi Jin, and Yihong Dong. 2025. Uncertainty-guided chain-of-thought for code generation with llms. arXiv preprint arXiv:2503.15341. — [arXiv:2503.15341](https://arxiv.org/abs/2503.15341) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mu-2026-self-reflective-generation-test-time/