Elamur Research⌕
☰
home / papers / 2026 / DeepTrans: Deep Reasoning Translation via Reinforcement Learning

DeepTrans: Deep Reasoning Translation via Reinforcement Learning

Paper · 2026 · Jiaan Wang, Fandong Meng and Jie Zhou

Summary

The paper asks how reinforcement learning can teach a reasoning LLM to translate literary text freely, conveying meaning and cultural nuance rather than rendering it word for word. Standard MT metrics such as BLEU, COMET and CometKiwi correlate poorly with human judgment in literary text, and trained reward models require costly preference annotations. The authors therefore use DeepSeek-v3 (671B) as an LLM judge with hand-designed scoring rubrics. It scores three things: whether the output follows the required format, the quality of the reasoning, and the quality of the final translation. Training uses only source sentences, so no reference translations or synthetic chain-of-thought data are needed. The resulting model, DeepTrans-7B (built on Qwen2.5-7B), improves English-to-Chinese literary translation by 16.3%. It beats QwQ-32B-preview and the SFT-trained DRT-7B. The work matters because it offers a cheap, annotation-free path to better free translation. It also documents failed reward designs and a "stylization" side effect, in which the reward model's preferences push outputs toward classical-style Chinese.

Key points

  • Output format and format reward: The model writes its reasoning inside <think>…</think> and then gives the translation. A regex checks the structure, and DeepSeek-v3 checks that no explanations leak into the translation.
  • Graded rewards: Reasoning is rated on a 3-point scale and translations on a 100-point scale. When the translation scale was narrowed to 3 or 5 points, results on the MetaphorTrans test set got worse, which suggests a wider score range helps the policy escape local optima.
  • Thought reward ablation: Dropping the thought reward, or replacing it with a minimum-length penalty (k = 10 tokens), produced generic, empty reasoning and lower scores on every metric.
  • Results: DeepTrans-7B gains 16.3% over its backbone and beats DRT-7B on automatic metrics and GPT-4o evaluation.
  • Limitations: Experiments cover only English→Chinese and 7B models, and the DeepSeek-v3 reward is computationally expensive. Code is available at github.com/krystalan/DRT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China134 works · 19,743 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Jiaan Wang, Fandong Meng and Jie Zhou. 2026. DeepTrans: Deep Reasoning Translation via Reinforcement Learning. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.65

@article{wang2026deeptrans,
  title = {{DeepTrans: Deep Reasoning Translation via Reinforcement Learning}},
  author = {Jiaan Wang, Fandong Meng and Jie Zhou},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.65},
  url = {https://doi.org/10.1162/tacl.a.65}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. find
  2. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations. Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris… — The Llama 3 Herd of Models
  3. Katharina Barbe. 1996. The dichotomy free and literal translation. Meta, 41(3):328–337. https://doi.org/10.7202/001968ar — [doi:10.7202/001968ar](https://doi.org/10.7202/001968ar) ↗
  4. Andong Chen, Yuchen Song, Wenxin Zhu, Kehai Chen, Muyun Yang, Tiejun Zhao, and Min zhang. 2025a. Evaluating o1-like llms: Unlocking reasoning for translation through comprehensive analysis. ArXiv preprint, abs/2502.11544v1. find
  5. Qi Chen, Oi Yee Kwong, and Jingbo Zhu. 2018. Detecting free translation in parallel corpora from attention scores. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation. Hong Kong. Association for Computational Linguistics. find
  6. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. 2025b. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. ArXiv preprint, abs/2503.09567v3.
  7. Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations. find
  8. Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. 2025. DeepRAG: Thinking to retrieval step by step for large language models. ArXiv preprint, abs/2502.01142v1. find
  9. Zhaopeng Feng, Shaosheng Cao, Jiahan Ren, Jiayuan Su, Ruizhe Chen, Yan Zhang, Zhe Xu, Yao Hu, Jian Wu, and Zuozhu Liu. 2025. MT-R1-Zero: Advancing llm-based machine translation via r1-zero-like reinforcement learning. ArXiv preprint, abs/2504.10160v1. https://doi.org/10.18653/v1/2025 .findings-emnlp… ↗
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Che… find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Junlong Li, J. L. C… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  12. Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.41 — Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist ↗
  13. Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1673–1681, Online. A… find
  14. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada. As…
  15. Minggui He, Yilun Liu, Shimin Tao, Yuanchang Luo, Hongyong Zeng, Chang Su, Li Zhang, Hongxia Ma, Daimeng Wei, Weibin Meng, et al. 2025. R1-T1: Fully incentivizing translation capability in llms via reasoning learning. ArXiv preprint, abs/2502.19735v2.
  16. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. find
  17. Jian Hu. 2025. Reinforce++: A simple and efficient approach for aligning large language models. ArXiv preprint, abs/2501.03262v1. find
  18. Daniil Larionov, Sotaro Takeshita, Ran Zhang, Yanran Chen, Christoph Leiter, Zhipin Wang, Christian Greisinger, and Steffen Eger. 2025. DeepSeek vs. o3-mini: How well can reasoning llms evaluate mt and summarization? ArXiv preprint, abs/2504.08120v1. find
  19. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training llms to reason and leverage search engines with reinforcement learning. ArXiv preprint, abs/2503.09516v3.
  20. Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020. Dynamic context selection for document-level neural machine translation via reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2242–2254, Online. Associatio… find
  21. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and others. 2024. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. ArXiv preprint, abs/2411.16594v6. find
  22. Xinzhe Li. 2025. A survey on LLM test-time compute via search: Tasks, LLM profiling, search algorithms, and relevant frameworks. Transactions on Machine Learning Research. find
  23. Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Proceedings of the Eighth Conference on Machine Translation, pages 419–451, Singapore. Association for Zhong-Zhi Li, Duzhen Zhang, Ming…
  24. OpenAI. 2024a. GPT-4o system card. ArXiv preprint, abs/2410.21276v1.
  25. OpenAI. 2024b. Learning to reason with large language models. https://openai.com/index/learning-to-reason-with -llms/ Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji,… ↗
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  27. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In International Conference on Learning Representations.
  28. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. https:/… find
  29. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. https://doi.org/10.186…
  30. Sinuo Liu, Chenyang Lyu, Minghao Wu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. New trends for modern machine translation with large reasoning models. ArXiv preprint, abs/2503.10351v1.
  31. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. find
  32. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  33. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Web-GPT: Browser-assi…
  34. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300v3.
  35. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a good NLG evaluator? A preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Lingu…
  36. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683–1692, Berlin, Germany. Associatio…
  37. Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. 2025. DRT: Deep reasoning translation via long chain-of-thought. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6770–6782, Vienna, Austria. Association for Computational Linguistics. https://doi.org /10.18653/v1/20… ↗
  38. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021. find
  39. Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3612–3621, Brussels, Belgium. Association for Computational Linguistic…
  40. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. 2025. Stop overthinking: A survey on efficient reasoning for large language models. Transactions on Machine Learning Research.
  41. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  42. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Yuan Wu, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng Ann Heng, Jife… find
  43. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… — [arXiv:2503.01067](https://arxiv.org/abs/2503.01067) find
  44. Qwen Team. 2025. QwQ-32B: Embracing the power of reinforcement learning. find
  45. Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, et al. 2024. Qwen2.5 technical report. ArXiv preprint, abs/2412.15115v2. find
  46. Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. 2025. DAPO: An open-source llm reinforcement learning system at scale. ArXiv preprint, abs/2503.14476v1.
  47. Nicolas Yax, Hernán Anlló, and Stefano Palminteri. 2024. Studying and improving reasoning in humans and machines. Communications Psychology, 2(1):51. https://doi.org/10.1038/s44271-024-00091-8, PubMed: 39242743 — [doi:10.1038/s44271-024-00091-8](https://doi.org/10.1038/s44271-024-00091-8) ↗
  48. Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. 2024. o1-Coder: an o1 replication for coding. ArXiv preprint, abs/2412.00154v2. — [arXiv:2412.00154](https://arxiv.org/abs/2412.00154) find
  49. Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-o1: Towards open reasoning models for open-ended solutions. ArXiv preprint, abs/2411.14405v2. find
  50. Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2024. Natural language reasoning, a survey. ACM Computing Surveys, 56(12):1–39. https://doi.org/10 .1145/3664194 find
  51. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thail…
  52. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan D… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-deeptrans-deep-reasoning-translation-reinforcement/