Elamur Research⌕
☰
home / papers / 2025 / DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning

DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning

Paper · 2025 · Hang Wu, Hongkai Chen, Yujun Cai et al.

Summary

Locating the on-screen element that a natural-language command refers to (GUI grounding) is difficult. High-resolution screenshots are crowded with distractors, and vision-language models tend to favour visible text over icons, even when an icon is the real target. DiMo-GUI is a training-free, plug-and-play inference procedure that wraps existing models. It runs grounding twice: once with a prompt that restricts attention to text elements and once with a prompt restricted to icons. Each run uses dynamic zooming: the model predicts a point, the image is cropped around that point to half its size, and prediction repeats until a stopping condition is met. The same model then decides which of the two candidates better fits the command. The authors integrate DiMo-GUI into several state-of-the-art GUI agents and test it on ScreenSpot-Pro and ScreenSpot-v2. Accuracy improves consistently, and the largest gains appear on high-resolution, cluttered screens. Because the method requires no retraining, extra data, or annotation, it offers a cheap way to make GUI agents more reliable.

Key points

  • The method has three steps: modality division (text vs. icon prompts), dynamic zooming with a halting mechanism to limit overthinking, and final candidate selection by the model itself.
  • In the ablation with OS-Atlas-7B, accuracy rose from 18.4% (vanilla) to 45.7% with dynamic grounding alone, 26.1% with modality division alone, and 49.7% with full DiMo-GUI.
  • Accuracy grows with more zoom iterations, from 18.4% up to about 48.9%, and gains level off after roughly three steps. The authors note that excessive zooming can remove useful context.
  • A case study shows the text bias in action: an "Edit" text label lures the model away from the correct icon. With an icon-only prompt, the same label instead serves as a helpful contextual cue.
  • The main limitation is the lack of backtracking or error correction, so early mistakes propagate. The authors propose tree- or graph-based search as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksSoftware · 92 worksControl and Systems Engineering · 45 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang and Yiwei Wang. 2025. DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1334

@inproceedings{wu2025dimo,
  title = {{DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning}},
  author = {Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang and Yiwei Wang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1334},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1334}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jinwoo Shin, and Kang Min Yoo. 2025a. ReGUIDE: Data efficient gui grounding via spatial reasoning and search. arXiv preprint arXiv:2505.15259. — [arXiv:2505.15259](https://arxiv.org/abs/2505.15259) find
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. Proceedings of the IEEE/CVF Conference on Computer Vis…
  3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
  4. Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack. 2025b. Revise: Learning to refine at test-time via intrinsic self-verification. arXiv preprint arXiv:2502.14565. — [arXiv:2502.14565](https://arxiv.org/abs/2502.14565) find
  5. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025. ScreenSpot-Pro: Gui grounding for professional high-resolution computer use. find
  6. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. 2023. Introducing our multimodal models. find
  7. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. ShowUI: One vision-language-action model for gui visual agent. find
  8. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
  9. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2024a. Grounding DINO: Marrying dino with grounded pre-training for open-set object detection.
  10. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. Annual Meeting of the Association for Computational Linguistics (ACL).
  11. Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. 2020. PP-OCR: A practical ultra lightweight ocr system. find
  12. Haonan Ge, Yiwei Wang, Ming-Hsuan Yang, and Yujun Cai. 2025. Mrfd: Multi-region fusion decoding with self-consistency for mitigating hallucinations in lvlms. arXiv preprint arXiv:2508.10264. find
  13. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. International Conference on Learning Representations (ICLR). find
  14. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. CogAgent: A visual language model for gui agents.
  15. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. 2024. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457.
  16. Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. 2024b. VisualAgentBench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327. — [arXiv:2408.06327](https://arxiv.org/abs/2408.06327) find
  17. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203. — [arXiv:2408.00203](https://arxiv.org/abs/2408.00203) find
  18. Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. 2025. Visual test-time scaling for gui agent grounding. arXiv preprint arXiv:2505.00684. — [arXiv:2505.00684](https://arxiv.org/abs/2505.00684) find
  19. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. WebGPT: Browser-assis…
  20. Anthony Nguyen. 2024. Improved gui grounding via iterative narrowing. arXiv preprint arXiv:2411.13591. — [arXiv:2411.13591](https://arxiv.org/abs/2411.13591) find
  21. OpenAI. 2023. Gpt-4 technical report.
  22. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. 2025. UI-TARS: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326.
  23. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
  24. Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, and Yueting Zhuang. 2025. Think Twice, Click Once: Enhancing gui grounding via fast and slow systems. arXiv preprint arXiv:2503.06470. — [arXiv:2503.06470](https://arxiv.org/abs/2503.06470) find
  25. Xingjian Tao, Yiwei Wang, Yujun Cai, Zhicheng Yang, and Jing Tang. 2025. Understanding gui agent localization biases through logit sharpness. arXiv preprint arXiv:2506.15425. find
  26. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024a. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. find
  27. Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. UFO: A ui-focused agent for windows os interaction.
  28. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. AppAgent: Multimodal agents as smartphone users. find
  29. Zhuosheng Zhang and Aston Zhang. 2023. You only look at screens: Multimodal chain-of-action agents. Annual Meeting of the Association for Computational Linguistics (ACL).
  30. Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinqlin Jia, et al. 2025. GUI-G1: Understanding r1-zero-like training for visual grounding in gui agents. arXiv preprint arXiv:2505.15810. — [arXiv:2505.15810](https://arxiv.org/abs/2505.15810) find
  31. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the…
  32. Penghao Wu and Saining Xie. 2024. V*: Guided visual search as a core mechanism in multimodal llms. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084–13094. find
  33. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. 2024. Os-atlas: A foundation action model for generalist gui agents. International Conference on Learning Representations (ICLR).
  34. Xiaobo Xia and Run Luo. 2025. GUI-R1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458. — [arXiv:2504.10458](https://arxiv.org/abs/2504.10458) find
  35. Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. 2024a. Agentoccam: A simple yet strong baseline for llm-based web agents. arXiv preprint arXiv:2410.13825. — [arXiv:2410.13825](https://arxiv.org/abs/2410.13825) find
  36. Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. 2024b. Aria-UI: Visual grounding for gui instructions. find
  37. Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. 2025. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. arXiv preprint arXiv:2505.12370. — [arXiv:2505.12370](https://arxiv.org/abs/2505.12370) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2025-dimo-gui-advancing-test-time/