Elamur Research⌕
☰
home / papers / 2026 / NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks

NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks

Paper · 2026 · Zhihao Luo, Wentao Yan, Jingyu Gong et al.

Summary

GUI agents and embodied navigation agents have mostly been built separately, each with its own datasets, action spaces and training recipes. This raises costs and limits out-of-domain (OOD) generalization. The authors note that both settings can be framed as Markov Decision Processes over egocentric visual observations. On that basis they build NaviMaster, a single policy for both.

The method has three components:

  • Visual-target trajectory collection: GUI and embodied trajectories are recast into one action space in which each step points to a visual target. GPT-4o adds a first-person reasoning thought to each step.
  • Unified reinforcement learning: one training framework covers the mixed data, uses task-specific advantage estimation, and conditions each prediction on prior reasoning and actions.
  • Distance-aware dense reward: predictions are scored by how close they land to the target point, instead of a binary hit-or-miss signal.

On OOD benchmarks, NaviMaster surpasses earlier agents across three areas: navigating GUIs, predicting spatial affordances, and ObjectNav-style embodied navigation. This suggests that training on both domains together yields more transferable spatial reasoning than training a specialist on either one.

Key points

  • Spatial affordance (Table 2): NaviMaster scores 77.34 on RoboReflt, 52.97 on Where2Place, 21.65 on RoboSpatial and 19.49 on RefSpatial. It beats RoboPoint-13B, SpaceLLaVA, GPT-4o and Qwen2.5VL-7B on all four.
  • Embodied navigation (ObjectNav under VLMNav): NaviMaster reaches roughly 33% SR and 12.60 SPL. The Qwen2.5VL-7B baseline gets 27.23 SR and 9.68 SPL. Variants trained only on GUI or only on embodied data score lower.
  • Data mixing ratio: a 5:5 GUI/embodied mix performs best. Even unbalanced mixes outperform single-domain training, and this holds at both 7k and 20k samples and across Qwen2.5VL-3B and Qwen2VL-7B backbones.
  • Visual-relation benchmark: the authors built a new 100-instance test from AndroidControl in which targets are described only relative to a landmark UI element. The mixed-data model clearly beats single-domain models and existing RFT agents on it.
  • Reward design: the dense reward (distance threshold 200) consistently beats a sparse binary reward (threshold 20) and gives a faster-rising reward curve. Code, data and checkpoints are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Zhihao Luo, Wentao Yan, Jingyu Gong, Min Wang, Zhizhong Zhang, Xuhong Wang, Yuan Xie and Xin Tan. 2026. NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1788

@inproceedings{luo2026navimaster,
  title = {{NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks}},
  author = {Zhihao Luo, Wentao Yan, Jingyu Gong, Min Wang, Zhizhong Zhang, Xuhong Wang, Yuan Xie and Xin Tan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1788},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1788}
}

References

49 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  2. Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. 2020. Objectnav revisited: On evaluation of embodied agents navigating to objects. Preprint, arXiv:2006.13171. — [arXiv:2006.13171](https://arxiv.org/abs/2006.13171) find
  3. Cong Chen, Kaixiang Ji, Hao Zhong, Muzhi Zhu, Anzhou Li, Guo Gan, Ziyuan Huang, Cheng Zou, Jiajia Liu, Jingdong Chen, Hao Chen, and Chunhua Shen. 2025a. Gui-shepherd: Reliable process reward and verification for long-sequence gui tasks. Preprint, arXiv:2509.23738. — [arXiv:2509.23738](https://arxiv.org/abs/2509.23738) find
  4. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, and 1 others. 2025b. Guicourse: From general vision language model to versatile gui agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics… find
  5. Matthew Foutter, Daniele Gammelli, Justin Kruger, Ethan Foss, Praneet Bhoj, Tommaso Guffanti, Simone D’Amico, and Marco Pavone. 2025. Spacellava: A vision-language model adapted to extraterrestrial applications. In 2025 IEEE Aerospace Conference, pages 1–23. IEEE. find
  6. Guo Gan, Yuxuan Ding, Cong Chen, Yuwei Ren, Yin Huang, and Hong Zhou. 2026. Android coach: Improve online agentic training efficiency with single state multiple actions. Preprint, arXiv:2604.07277.
  7. Longxi Gao, Li Zhang, Pengzhi Gao, Wei Liu, Jian Luan, and Mengwei Xu. 2026a. GUI-shift: Enhancing VLM-based GUI agents through self-supervised reinforcement learning. In The Fourteenth International Conference on Learning Representations. find
  8. Peng Gao, Peng Wang, Feng Gao, Fei Wang, and Ruyue Yuan. 2024. Vision-language navigation with embodied intelligence: A survey. Preprint, arXiv:2402.14304. — [arXiv:2402.14304](https://arxiv.org/abs/2402.14304) find
  9. Zhenkun Gao, Xuhong Wang, Xin Tan, and Yuan Xie. 2026b. Tpru: Advancing temporal and procedural understanding in large multimodal models. In The Fourteenth International Conference on Learning Representations. find
  10. Dylan Goetting, Himanshu Gaurav Singh, and Antonio Loquercio. 2025. End-to-end navigation with vision-language models: Transforming spatial reasoning into question-answering. In International Conference on Neuro-symbolic Systems, pages 22– 35. PMLR. 38603 9 find
  11. Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107. find
  12. Yining Hong, Rui Sun, Bingxuan Li, Xingcheng Yao, Maxine Wu, Alexander Chien, Da Yin, Ying Nian Wu, Zhecan Wang, and Kai-Wei Chang. 2025. Embodied web agents: Bridging physical-digital realms for integrated agent intelligence. In The Thirty-ninth Annual Conference on Neural Information Processing Sy… find
  13. Yuzhou Ji, Qijian Tian, He Zhu, Xiaoqi Jiang, Guangzhi Cao, Lizhuang Ma, Yuan Xie, and Xin Tan. 2026. S2d: Sparse to dense lifting for 3d reconstruction with minimal inputs. find
  14. Yuzhou Ji, He Zhu, Junshu Tang, Wuyi Liu, Zhizhong Zhang, Xin Tan, and Yuan Xie. 2025. Fastlgs: Speeding up language embedded gaussians with feature grid mapping. In Proceedings of the AAAI conference on artificial intelligence, volume 39, pages 3922– 3930. find
  15. Yizhang Jin, Jian Li, Tianjun Gu, Yexin Liu, Bo Zhao, Jinxiang Lai, Zhenye Gan, Yabiao Wang, Chengjie Wang, Xin Tan, and Lizhuang Ma. 2025. Efficient multimodal large language models: a survey. Visual Intelligence, 3(1).
  16. Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2024. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pages 161– 178. Springer. find
  17. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  18. LinFeng Li, Jian Zhao, Yuan Xie, Xin Tan, and Xuelong Li. 2026. Compassnav: Steering from path imitation to decision understanding in navigation. In The Fourteenth International Conference on Learning Representations. find
  19. Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130–92154. find
  20. Shuquan Lian, Yuhang Wu, Jia Ma, Yifan Ding, Zihan Song, Bingqi Chen, Xiawu Zheng, and Hui Li. 2025. Ui-agile: Advancing gui agents with effective reinforcement learning and precise inference-time grounding. Preprint, arXiv:2507.22025. — [arXiv:2507.22025](https://arxiv.org/abs/2507.22025) find
  21. Bingqian Lin, Yunshuang Nie, Khun Loun Zai, Ziming Wei, Mingfei Han, Rongtao Xu, Minzhe Niu, Jianhua Han, Liang Lin, Cewu Lu, and Xiaodan Liang. 2025. Evolvenav: Self-improving embodied reasoning for llm-based vision-language navigation. Preprint, arXiv:2506.01551. — [arXiv:2506.01551](https://arxiv.org/abs/2506.01551) find
  22. Bangwei Liu, Yicheng Bao, Shaohui Lin, Xuhong Wang, Xin Tan, Yingchun Wang, Yuan Xie, and Chaochao Lu. 2025a. Idmr: Towards instance-driven precise visual correspondence in multimodal retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6320–6329. find
  23. Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. 2025b. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. Preprint, arXiv:2504.14239. — [arXiv:2504.14239](https://arxiv.org/abs/2504.14239) find
  24. Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. 2025c. Visual-rft: Visual reinforcement fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2034–2044. find
  25. Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. 2025a. Guiodyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pag… find
  26. Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. 2023. Vl-grasp: a 6-dof interactive grasp policy for language-oriented objects in cluttered indoor scenes. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 976–983. IEEE. find
  27. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. 2025b. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. Preprint, arXiv:2503.21620. — [arXiv:2503.21620](https://arxiv.org/abs/2503.21620) find
  28. Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. 2025. Gui-r1 : A generalist r1-style vision-language action model for gui agents. Preprint, arXiv:2504.10458. — [arXiv:2504.10458](https://arxiv.org/abs/2504.10458) find
  29. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  30. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025. Ui-tars: Pioneering automated gui interaction with…
  31. Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, and 1 others. 2021. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In Thirty-fif… find
  32. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap. 2023. An-droidinthewild: A large-scale dataset for android device control. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  33. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, and 1 others. 2019. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339… find
  34. Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. 2025. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15768–15780. find
  35. Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. 2025. Drivingforward: Feed-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7374–7382. find
  36. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  37. Sen Wang, Bangwei Liu, Zhenkun Gao, Lizhuang Ma, Xuhong Wang, Yuan Xie, and Xin Tan. 2026. Explore with long-term memory: A benchmark and multimodal llm-based reinforcement learning framework for embodied exploration. find
  38. Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, and Jianye Hao. 2025. Gui agents with foundation models: A comprehensive survey. Preprint, arXiv:2411.04890. — [arXiv:2411.04890](https://arxiv.org/abs/2411.04890) find
  39. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. 2025. OS-ATLAS: Foundation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Representations. find
  40. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Aguvis: Unified pure vision agents for autonomous gui interaction. In International Conference on Machine Learning, pages 69772–69805. PMLR. find
  41. Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, and 1 others. 2023. Habitat-matterport 3d semantics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog… find
  42. Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2025a. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643. find
  43. Longrong Yang, Zhixiong Zeng, Yufeng Zhong, Huang Jing, Liming Zheng, Lei Chen, Haibo Qiu, Zequn Qin, Lin Ma, and Xi Li. 2026. Omniactor: A generalist GUI and embodied agent for 2d&3d worlds. In The Fourteenth International Conference on Learning Representations. find
  44. Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, and 1 others. 2025b. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. In International Conference… find
  45. Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. 2025. Robopoint: A vision-language model for spatial affordance prediction in robotics. In Conference on Robot Learning, pages 4005–4020. PMLR. find
  46. Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024a. Android in the zoo: Chain-of-action-thought for gui agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12016–12031. find
  47. Lechao Zhang, Haoran Xu, Jingyu Gong, Xuhong Wang, Yuan Xie, and Xin Tan. 2026. World2minecraft: Occupancy-driven simulated scenes construction. In The Fourteenth International Conference on Learning Representations. find
  48. Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. 2024b. Llamatouch: A faithful and scalable 38605 11 testbed for mobile ui task automation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–13. find
  49. Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. 2025a. Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-navimaster-learning-unified-policy-gui/