Elamur Research⌕
☰
home / papers / 2025 / UFO: A UI-Focused Agent for Windows OS Interaction

UFO: A UI-Focused Agent for Windows OS Interaction

Paper · 2025 · Chaoyun Zhang, Liqun Li, Shilin He et al.

Summary

Windows is a hard environment for vision-language agents. Its interfaces are dense, its screens are large, and everyday requests often require moving between several programs. The Microsoft authors present UFO, an agent that turns natural-language instructions into executed GUI actions on Windows. It relies on two cooperating agents:

  • HostAgent examines desktop screenshots and the list of active applications, drafts a global plan and assigns each sub-task to a program.
  • AppAgent works step by step inside the chosen application, using previous, clean and SoM-annotated screenshots.

A Control Automator built on Windows UIA and pywinauto finds control elements and converts the agents' decisions into real clicks and text entry. The system also lets users add custom actions and asks for confirmation before sensitive steps. The authors evaluated UFO on WindowsBench (50 tasks spanning 9 widely used applications) with quantitative metrics and case studies. UFO clearly outperformed competing GUI agents, and it handled cross-application work well, such as assembling an Outlook email from Word, image and PowerPoint content. The authors present it as the first UI agent designed specifically for Windows, and the code is open source.

Key points

  • The architecture is hierarchical and dual-agent. HostAgent handles task decomposition, application selection and switching between applications, while AppAgent executes actions iteratively within a single application.
  • Both agents prompt GPT-V with in-context examples and Chain-of-Thought style reasoning. The outputs include observations, thoughts, the selected application or control, a status of CONTINUE or FINISH, and comments for the user.
  • Controls are labelled using the Set-of-Mark (SoM) method. Grounded execution is limited to what pywinauto and Windows UIA support; the authors plan future backends such as the Win32 API or visual grounding models like CogAgent.
  • The WindowsBench benchmark contains 50 requests across 9 popular Windows applications. A safeguard requires user confirmation for risky actions, such as clicking "Send."
  • The stated limitations are difficulty with unfamiliar or niche applications and interruptions when the user operates the mouse or keyboard during a run. Code is available at github.com/microsoft/UFO.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHuman-Computer Interaction · 141 works

Cite

Cite as Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang and Qi Zhang. 2025. UFO: A UI-Focused Agent for Windows OS Interaction. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.26

@inproceedings{zhang2025ufo,
  title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
  author = {Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang and Qi Zhang},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.26},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.26}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akinlolu Adekotujo, Adedoyin Odumabo, Ademola Adedokun, and Olukayode Aiyeniko. 2020. A comparative study of operating systems: Case of windows, unix, linux, mac, android and ios. International Journal of Computer Applications, 176(39):16– 23. find
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
  3. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
  4. Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, et al. 2023. TaskWeaver: A code-first agent framework. arXiv preprint arXiv:2311.17541. — [arXiv:2311.17541](https://arxiv.org/abs/2311.17541) find
  5. HE Bim and WANG Min-shuai. 2014. Application of pywinauto in software performance test. Computer and Modernization, (8):135. find
  6. Rudolf Ramler, Thomas Wetzlmaier, and Robert Hoschek. 2018. Gui scalability issues of windows desktop applications and how to find them. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops, pages 63–67. find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  8. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.…
  9. Yuhang Chen, Chaoyun Zhang, Minghua Ma, Yudong Liu, Ruomeng Ding, Bowen Li, Shilin He, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Imdiffusion: Imputed diffusion models for multivariate time series anomaly detection. arXiv preprint arXiv:2307.00754. — [arXiv:2307.00754](https://arxiv.org/abs/2307.00754) find
  10. Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366.
  11. Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254. find
  12. Duong Tran Dinh, Pham Ngoc Hung, and Tung Nguyen Duy. 2018. A method for automated user interface testing of windows-based applications. In Proceedings of the 9th International Symposium on Information and Communication Technology, pages 337– 343. find
  13. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023. CogAgent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914.
  14. Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. 2024. Towards general computer control: A multimodal agent for red dead redemption ii as a case study. arXiv preprint arXiv:2403.03186. — [arXiv:2403.03186](https://arxiv.org/abs/2403.03186) find
  15. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158. — [arXiv:2401.16158](https://arxiv.org/abs/2401.16158) find
  16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  17. Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, et al. 2023. Xpert: Empowering incident management with query recommendations via large language models. arXiv preprint arXiv:2312.11988. — [arXiv:2312.11988](https://arxiv.org/abs/2312.11988) find
  18. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026.
  19. William Stallings. 2005. The windows operating system. Operating Systems: Internals and Design Principles. find
  20. Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, et al. 2023. Can GPT-4V (ision) serve medical applications? case studies on GPT-4V for multimodal medical diagnosis. arXiv preprint arXiv:2310.09909. — [arXiv:2310.09909](https://arxiv.org/abs/2310.09909) find
  21. Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. 2024. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456. find
  22. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023a. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. find
  23. Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023b. AppAgent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771. find
  24. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023c. The dawn of lmms: Preliminary explorations with GPT-4V (ision). arXiv preprint arXiv:2309.17421, 9(1):1.
  25. Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. 2023. GPT-4V (ision) as a generalist evaluator for vision-language tasks. arXiv preprint arXiv:2311.01361. — [arXiv:2311.01361](https://arxiv.org/abs/2311.01361) find
  26. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. GPT-4V (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-ufo-ui-focused-agent-windows/