Elamur Research⌕
☰
home / papers / 2026 / From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

Paper · 2026 · Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang et al.

Summary

End-to-end GUI agents, which map screenshots directly to actions, are simpler to deploy than framework-based computer-use systems but score noticeably lower on OSWorld-Verified. Interactive, verifiable tasks are scarce, so only a small set of expert trajectories exists, mostly produced by framework agents. The authors study how RLVR can make use of these traces. Feeding them directly into on-policy training tends to fail. Framework traces combine several roles and tool-level actions, and even converted versions remain far from the learner's own distribution.

BEPA addresses this in two stages. LEVEL-1 has the base policy re-execute extracted expert plans, which yields trajectories the policy can actually reach. LEVEL-2 keeps a per-task cache that is refreshed with the agent's own successes. The cache is fed into GRPO only when every rollout in a group fails.

With UITARS1.5-7B, BEPA raises OSWorld-Verified success from 22.87% to 32.13% and also improves results on a held-out split, MMBench-GUI and Online-Mind2Web. The main lesson is that heterogeneous expert data works best when it is gradually absorbed into the learner's distribution rather than mixed in statically.

Key points

  • Main results: UITARS1.5-7B improves from 22.87% to 32.13% on OSWorld-Verified, while its success on a strictly held-out split climbs from 5.74% to 10.30%. For context, framework agents such as Agent S2 reach 33.00–42.40%.
  • Expert sources and baselines: expert traces come from Agent S2 and GUI-Owl. Static mixing baselines (LUFFY, Replacement, SFT+RL) can perform worse than plain GRPO.
  • Distribution alignment: the gap between guidance and on-policy behaviour is measured by JS divergence. It falls from 0.2040 (raw traces) to 0.1676 (converted), 0.1525 (LEVEL-1) and 0.0366 (LEVEL-2). The share of low-probability tokens drops from 58.33% (raw) to 10.53% (LEVEL-1).
  • Ablation (expert-covered / held-out success):
    • Replacement: 18.52% / 1.29%
    • LEVEL-1: 25.93% / 5.05%
    • LEVEL-2: 29.18% / 7.48%
    • Full BEPA: 35.19% / 10.30%
  • Availability: code and data are released in the GitHub repository LEON-gittech/Verl_GUI.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢Microsoft Research AsiaBeijing · China73 works · 60,539 citations🎓Nanjing UniversityNanjing · China74 works · 7,814 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 works

Cite

Cite as Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Zhuzhong Qian and Yan Lu. 2026. From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1082

@inproceedings{wang2026off,
  title = {{From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation}},
  author = {Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Zhuzhong Qian and Yan Lu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1082},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1082}
}

References

55 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. 2024. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032. — [arXiv:2407.13032](https://arxiv.org/abs/2407.13032) find
  2. Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. 2025. Agent s: An open agentic framework that uses computers like a human. In International Conference on Learning Representations (ICLR). find
  3. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. In Annual Meeting of the Association for Computational Linguistics (AC… find
  4. Adept AI. 2022. Act-1: Transformer for actions. https://www.adept.ai/blog/act-1. Blog post. ↗
  5. LaVague AI. 2025. Lavague: Large action model framework. https://github.com/lavague-ai/ LaVague. GitHub repository. ↗
  6. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  7. Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. 2025. Reinforcement learning for long-horizon interactive llm agents. Preprint, arXiv:2502.01600. — [arXiv:2502.01600](https://arxiv.org/abs/2502.01600) find
  8. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, R. Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 179 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  9. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023a. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS).
  10. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023b. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS).
  11. Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, and 1 others. 2024. AssistGUI: Task-oriented pc graphical user interface automation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  12. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for gui agents. In International Conference on Learning Representations (ICLR). find
  13. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, and 178 others. 2025. Seed1.5-vl technical report. arXiv preprint arXiv:250…
  14. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, X. Zhang, and H. Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.
  15. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. 2025. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004. Under review at ICLR 2026. — [arXiv:2508.05004](https://arxiv.org/abs/2508.05004) find
  16. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8925–8948. Association for Computational Linguistics.
  17. Rong Jin, Xiangheng Li, and Tong Zhang. 2024. On stationary point convergence of ppo-clip. In Proceedings of the International Conference on Learning Representations, ICLR. find
  18. Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3-4):293–321. find
  19. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR). find
  20. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. Preprint, arXiv:2503.20783.
  21. I. Loshchilov and F. Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). find
  22. Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. 2025. Arpo:end-to-end policy optimization for gui agents with experience replay. Preprint, arXiv:2505.16282. — [arXiv:2505.16282](https://arxiv.org/abs/2505.16282) find
  23. Jiantao Mei, Yu Bai, Chi Jin, and Jason Lee. 2020. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, ICML. find
  24. M. Menéndez, J. A. Pardo, L. Pardo, and M. Pardo. 1997. The jensen-shannon divergence. Journal of The Franklin Institute-engineering and Applied Mathematics, 334:307–318. find
  25. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, and 1 others. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529– 533. find
  26. Magnus Müller and Gregor Žunič. 2024. Browser use: Enable ai to control your browser. https://github.com/browser-use/browser-use. GitHub repository. ↗
  27. Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Md. Mehrab Tanjim, and 10 others. 2024. Gui agents: A survey. In Findings of the Associ… find
  28. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, P. Christiano, Jan Leike, and Ryan J. Lowe. 2022. T… — Training language models to follow instructions with human feedback
  29. Rui Qian, Xin Yin, Chuanhang Deng, Zhiyuan Peng, Jian Xiong, Wei Zhai, and Dejing Dou. 2025. Uground: Towards unified visual grounding with unrolled transformers. arXiv preprint arXiv:2510.03853. — [arXiv:2510.03853](https://arxiv.org/abs/2510.03853) find
  30. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025. Ui-tars: Pioneering automated gui interaction with…
  31. Rafael Rafailov, Archit Sharma, E. Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS).
  32. Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alexander Smola. 2016. Stochastic variance reduction for nonconvex optimization. In Proceedings of the 33rd International Conference on Machine Learning, ICML, pages 314–323. PMLR. find
  33. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2016. Prioritized experience replay. In 4th International Conference on Learning Representations. find
  34. John Schulman, S. Levine, P. Abbeel, Michael I. Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning (ICML). find
  35. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  36. Zhihong Shao, Peiyi Wang, Qihao Zhu, R. Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  37. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, Rotterdam, The Netherlands. ACM.
  38. Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Taiwei Shi, Ran Xu, and Caiming Xiong. 2025. Coact-1: Computer-using agents with coding as actions. arXiv preprint arXiv:2508.03923. — [arXiv:2508.03923](https://arxiv.org/abs/2508.03923) find
  39. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Bo Li, Chen Dun, Chong Liu, and 87 others. 2025a. Ui-tars-2 technical report: Advancing gui a…
  40. Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, and Ruiming Tang. 2024. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890. — [arXiv:2411.04890](https://arxiv.org/abs/2411.04890) find
  41. Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Bo Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, and 23 others. 2025b. Opencua: Open foundations for computer-use a… find
  42. Xuehui Wang, Zhenyu Wu, Jingjing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, and 9 others. 2025c. Mmbench-gui: Hierarchical multi-platform evalua… — [arXiv:2507.19478](https://arxiv.org/abs/2507.19478) find
  43. Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Scaling computer-use grounding via user interface decomposition and synthesis. Preprint, arXiv:2505.13227. find
  44. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, T. Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real…
  45. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382. find
  46. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945. find
  47. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. 2025. Mobile-agent-v3: Fundamental agents for gui automation. Preprint, arXiv:2508.15144. — [arXiv:2508.15144](https://arxiv.org/abs/2508.15144) find
  48. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, and 16 others. 2025. Dapo: An open-source llm reinforcement learning sys…
  49. Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Ming-Jie Ma, Qingwei Lin, S. Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279. — [arXiv:2411.18279](https://arxiv.org/abs/2411.18279) find
  50. Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. 2025. Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning. arXiv preprint arXiv:2506.17211. — [arXiv:2506.17211](https://arxiv.org/abs/2506.17211) find
  51. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. In Proceedings of the 41st International Conference on Machine Learning (ICML).
  52. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR).
  53. Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and G. Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find
  54. Wei Xiong, Hanze Dong, Chen Ye, Han Zhong, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference on Machine Learning (ICML). find
  55. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Aguvis: Unified pure vision agents for autonomous gui interaction. In International Conference on Machine Learning (ICML). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-off-policy-policy-enhancing-gui/