Elamur Research⌕
☰
home / papers / 2026 / LEPO: Latent Reasoning Policy Optimization for Large Language Models

LEPO: Latent Reasoning Policy Optimization for Large Language Models

Paper · 2026 · Yuyan Zhou, Jiarui Yu, Hande Dong et al.

Summary

Latent reasoning lets LLMs think with continuous representations, such as probability-weighted vocabulary embeddings, instead of committing to one token per step. Its drawback is determinism: the same prompt always produces the same latent trajectory, so the model cannot explore alternative reasoning paths, and reinforcement learning has little to work with. The authors address this by adding Gumbel noise to the vocabulary distribution through Gumbel-Softmax. This approximates multinomial sampling while keeping the latent token a soft distribution.

On this basis they build LEPO, an RL framework that works in two stages:

  • Rollout: the model generates a fixed number of noisy latent tokens, then writes a readable discrete answer.
  • Optimization: a single GRPO-style objective with group-normalized binary rewards updates both parts. Latent tokens are treated as soft targets and answer tokens as hard targets.

Pilot experiments on Qwen2.5-7B with MATH500 show that stochastic latent reasoning raises entropy and pass@32. It also turns many near-unsolvable problems into moderately solvable ones, which supplies denser reward signals. The authors report that LEPO beats discrete and latent RL baselines on math and general benchmarks. This suggests continuous reasoning can be trained with RL rather than only applied at inference time.

Key points

  • Gumbel-Softmax noise is controlled by a temperature τ_g. Low values push the latent token toward one-hot, and high values push it toward uniform. Both τ_g and the latent length L_g are analyzed as hyperparameters.
  • Using Qwen2.5-7B on MATH500, the stochastic latent setup moves problems from the 0.0–0.2 base-accuracy bin into the 0.2–0.5 range. The largest RL gains occur on these moderate-accuracy problems.
  • Generation is "latent-first, discrete-later," so only the final answer is human-readable. Latent steps can be roughly interpreted by decoding their top-1 token.
  • In the reported training curves, Gumbel noise converges better than GRPO and than variants using Gaussian or Dirichlet noise. A GSM8k example shows LEPO solving a profit problem that both GRPO and untrained latent CoT get wrong.
  • Code is released at github.com/YuyanZhou/lepo. A noted limitation is that latent and discrete tokens cannot yet be interleaved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Yuyan Zhou, Jiarui Yu, Hande Dong, Zhezheng Hao, Hong Wang, Jianqing Zhang and Qiang Lin. 2026. LEPO: Latent Reasoning Policy Optimization for Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.707

@inproceedings{zhou2026lepo,
  title = {{LEPO: Latent Reasoning Policy Optimization for Large Language Models}},
  author = {Yuyan Zhou, Jiarui Yu, Hande Dong, Zhezheng Hao, Hong Wang, Jianqing Zhang and Qiang Lin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.707},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.707}
}

References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Natasha Butt, Ariel Kwiatkowski, Ismail Labiad, Julia Kempe, and Yann Ollivier. 2025. Soft tokens, hard truths. arXiv preprint arXiv:2509.19170. — [arXiv:2509.19170](https://arxiv.org/abs/2509.19170) find
  2. Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13171.
  3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1.
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  5. Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. 2025. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171. find
  6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  7. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
  8. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  9. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  10. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. 2025. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074.
  11. Dachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan, Leyan Pan, Wenke Lee, and Wen Xiao. 2025. Swireasoning: Switch-thinking in latent and explicit for pareto-superior reasoning llms. arXiv preprint arXiv:2510.05069. — [arXiv:2510.05069](https://arxiv.org/abs/2510.05069) find
  12. Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. 2025. Llm pre-training with continuous concepts. arXiv preprint arXiv:2502.08524. — [arXiv:2502.08524](https://arxiv.org/abs/2502.08524) find
  13. Wenhui Tan, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, and Ruihua Song. 2025. Think silently, think fast: Dynamic latent compression of llm reasoning chains. arXiv preprint arXiv:2505.16552.
  14. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl. ↗
  15. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769. find
  16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  18. Chünhung Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, and Hua Wu. 2025. Llms are single-threaded reasoners: Demystifying the working mechanism of soft thinking. arXiv preprint arXiv:2508.03440. — [arXiv:2508.03440](https://arxiv.org/abs/2508.03440) find
  19. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  20. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  21. Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, and Dong Wang. 2025. Hybrid latent reasoning via reinforcement learning. arXiv preprint arXiv:2505.18454. — [arXiv:2505.18454](https://arxiv.org/abs/2505.18454) find
  22. Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. 2025. Soft thinking: Unlocking the reasoning potential of llms in continuous concept space. arXiv preprint arXiv:2505.15778.
  23. Zhi Zheng and Wee Sun Lee. 2025. Soft-grpo: Surpassing discrete-token llm reinforcement learning via gumbel-reparameterized soft-thinking policy optimization. arXiv preprint arXiv:2511.06411. — [arXiv:2511.06411](https://arxiv.org/abs/2511.06411) find
  24. Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. 2025a. Reasoning by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint arXiv:2505.12514. — [arXiv:2505.12514](https://arxiv.org/abs/2505.12514) find
  25. Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, and 1 others. 2025b. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.
  26. Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, and Jianfeng Gao. 2025. Text generation beyond discrete token sampling. arXiv preprint arXiv:2505.14827. — [arXiv:2505.14827](https://arxiv.org/abs/2505.14827) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-lepo-latent-reasoning-policy-optimization/