Elamur Research⌕
☰
home / papers / 2026 / On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning

On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning

Paper · 2026 · Di Wu, Jiaxin Fan, Chloe Gu et al.

Summary

Vision-language models handle static perception well but break down when they must plan many actions in an interactive environment. Online reinforcement learning could address this, but simulator interaction is expensive, and episode-level success signals are sparse. The authors propose ORBIT, which avoids environment feedback during training. It converts expert trajectories from ALFRED into training states and has the current policy generate the full remaining action sequence for each state. Each generated sequence is scored by how long its prefix agrees with the expert plan, and the policy is updated with GRPO. Training therefore becomes a single-step, contextual-bandit problem that is still on-policy. The authors also prove that the resulting policy stays within a bounded distance of one trained with real online interaction. On EmbodiedBench, ORBIT achieves the best EB-ALFRED result reported. It beats closed-source models and online-RL methods while needing about a quarter of their training time. This makes it a cheaper and more stable way to post-train embodied planners.

Key points

  • ORBIT reaches a 72.4% average success rate on EB-ALFRED, compared with 65.2% for Claude-3.5-Sonnet and 67.2% for the strongest online-RL baseline.
  • The training data consists of 43,898 samples derived from ALFRED. The accuracy reward is the quadratic prefix score n(n+1)/k(k+1), plus a format reward capped at 0.5. This reward beat a linear variant as well as embedding-based alternatives built on Qwen3-Embedding-0.8B, which scored plans by either OT distance or cosine similarity.
  • Efficiency gains:
    • Each update takes about 2 minutes, versus 12–15 minutes for online RL.
    • Full training converges in 8–10 hours, versus 36–40 hours.
    • At evaluation, tasks finish in 4.17 steps on average, versus 13.76 for single-step online policies.
  • A pipeline of SFT followed by RFT generalizes best to unseen environments, while RFT alone performs poorly. Scores on SpatialEval do not degrade and partly improve.
  • The method tolerates imperfect data:
    • With 10% of trajectories perturbed, the score is 69.6%; with 20% perturbed, it is 64.0%.
    • Using 50% of the data gives 68.4%; using 20% gives 65.2%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Duke UniversityDurham · United States37 works · 15,861 citations🎓Tongji UniversityShanghai · China33 works · 2,993 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksEngineering · 188 worksControl and Systems Engineering · 62 works

Cite

Cite as Di Wu, Jiaxin Fan, Chloe Gu, Guanbo Wang, Wei Yin, Wenhao Li and Bo Jin. 2026. On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1822

@inproceedings{wu2026policy,
  title = {{On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning}},
  author = {Di Wu, Jiaxin Fan, Chloe Gu, Guanbo Wang, Wei Yin, Wenhao Li and Bo Jin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1822},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1822}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741.
  2. Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, and Ian Reid. SayPlan: Grounding large language models using 3d scene graphs for scalable robot task planning. find
  3. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627– 635. JMLR Workshop and Conference Proceedi… find
  4. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, and 1 others. 2019. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339…
  5. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  6. Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, and 1 others. 2025. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615.
  7. Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. 2025. Rl’s razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.04259. — [arXiv:2509.04259](https://arxiv.org/abs/2509.04259) find
  8. Junhao Shi, Zhaoye Fei, Siyin Wang, Qipeng Guo, Jingjing Gong, and Xipeng QIu. 2025a. World-aware planning narratives enhance large vision-language model planner. arXiv preprint arXiv:2506.21230. — [arXiv:2506.21230](https://arxiv.org/abs/2506.21230) find
  9. Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, and 1 others. 2025b. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. arXiv preprint arXiv:2502.19417. — [arXiv:2502.19417](https://arxiv.org/abs/2502.19417) find
  10. Suyeon Shin, Sujin jeon, Junghyun Kim, Gi-Cheon Kang, and Byoung-Tak Zhang. Socratic planner: Inquiry-based zero-shot planning for embodied instruction following. Preprint, arxiv:2404.15190 [cs]. — [arXiv:2404.15190](https://arxiv.org/abs/2404.15190) find
  11. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page…
  12. Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, and 1 others. 2025. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09… — [arXiv:2502.09560](https://arxiv.org/abs/2502.09560) find
  13. Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in PDDL domains with pre-trained large language models. 38(18):20256–20264. find
  14. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387.
  15. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models. Preprint, arxiv:2209.11302 [cs]. Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo…
  16. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502. — [arXiv:2403.02502](https://arxiv.org/abs/2403.02502) find
  17. Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. 2024. Robotic control via embodied chain-of-thought reasoning. In 8th Annual Conference on Robot Learning. find
  18. Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. 2025. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752. — [arXiv:2503.20752](https://arxiv.org/abs/2503.20752) find
  19. Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Yang Katie Zhao, and Mingyi Hong. 2025. Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment. In ICML 2025 Workshop on Computer Use Agents. find
  20. Qwen Team. 2025. Qwen2. 5-vl technical report. — Qwen2.5-VL Technical Report
  21. Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, and Jianyu Chen. 2024a. Hirt: Enhancing robotic control with hierarchical robot transformers. arXiv preprint arXiv:2410.05273. — [arXiv:2410.05273](https://arxiv.org/abs/2410.05273) find
  22. Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. 2024. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392– 75421. find
  23. Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, and 1 others. 2025. Vagen: Reinforcing world model reasoning for multi-turn vlm agents. arXiv preprint arXiv:2510.16907. find
  24. Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, and Xipeng Qiu. World modeling makes a better planner: Dual preference optimization for embodied task planning. Preprint, arxiv:2503.10480 [cs]. find
  25. Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. 2023. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848. — [arXiv:2307.01848](https://arxiv.org/abs/2307.01848) find
  26. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, and 1 others. 2025. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101.
  27. Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, and Jian Tang. 2024. A survey on robotics with foundation models: toward embodied ai. arXiv preprint arXiv:2402.02385. — [arXiv:2402.02385](https://arxiv.org/abs/2402.02385) find
  28. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024b. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  29. Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. 2025a. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937. — [arXiv:2503.12937](https://arxiv.org/abs/2503.12937) find
  30. Wenqi Zhang, Mengna Wang, Gangao Liu, Xu Huixin, Yiwei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, and 1 others. 2025b. Embodied-reasoner: Synergizing visual search, reasoning, and action for embodied interactive tasks. arXiv preprint arXiv:2503.21696.
  31. Baining Zhao, Ziyou Wang, Jianjie Fang, Chen Gao, Fanhang Man, Jinqiang Cui, Xin Wang, Xinlei Chen, Yong Li, and Wenwu Zhu. 2025. Embodied-r: Collaborative framework for activating embodied spatial reasoning in foundation models via reinforcement learning. arXiv preprint arXiv:2504.12680. — [arXiv:2504.12680](https://arxiv.org/abs/2504.12680) find
  32. Hanqing Zhu, Zhenyu Zhang, Hanxian Huang, DiJia Su, Zechun Liu, Jiawei Zhao, Igor Fedorov, Hamed Pirsiavash, Zhizhou Sha, Jinwon Lee, and 1 others. 2025. The path not taken: Rlvr provably learns off the principals. arXiv preprint arXiv:2511.08567. — [arXiv:2511.08567](https://arxiv.org/abs/2511.08567) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-policy-reinforcement-fine-tuning-offline/