Elamur Research⌕
☰
home / papers / 2026 / Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study

Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study

Paper · 2026 · Li Zhang, Longxi Gao and Mengwei Xu

Summary

The paper asks whether vision-language models with built-in chain-of-thought reasoning make better mobile GUI agents, meaning systems that operate a smartphone from natural-language instructions. The authors compared six base and reasoning model pairs: the commercial Gemini 2.0 Flash and Claude 3.7 Sonnet, and the open-source AgentCPM-GUI-8B, GLM-4.5V, Qwen3-VL-4B and Qwen3-VL-8B. They tested several agent configurations on two static benchmarks, ScreenSpot and AndroidControl, and on the interactive AndroidWorld environment.

Overall, reasoning added little and sometimes lowered accuracy. The base and reasoning variants also failed on different tasks, so the gains and losses largely cancelled out. Manual error analysis traced the problems to two sources:

  • Benchmark weaknesses: ambiguous instructions, and static data that accepts only one correct action.
  • Model weaknesses: misreading screen details, and final answers that contradict the preceding reasoning.

Reasoning also multiplied output tokens, which raised latency and cost. The study matters because it questions the assumption that reasoning is always worth enabling. It argues for interactive evaluation, GUI-specific training, and adaptive or budgeted reasoning.

Key points

  • AndroidControl: Gemini Thinking reached 54.4% action accuracy, only 0.8 points above its base model. On ScreenSpot grounding, only Claude Thinking with normalized center-point output improved; other setups lost up to 29.7%.
  • AndroidWorld (116 tasks): Claude Thinking with set-of-mark prompting completed 64.7% of tasks, 6.3 points above its base model. The Gemini and Qwen3-VL reasoning variants performed slightly worse than their base versions.
  • Different failures: Gemini Thinking failed 36% (ScreenSpot), 9% (AndroidControl) and 12% (AndroidWorld) of the tasks its base model solved. It also solved up to 10% of the tasks the base model missed.
  • Cost: Output tokens rose by 3.11× to 14.78×. On ScreenSpot, the average grew from 37.6 to 238.5 tokens.
  • Reasoning budget: In a budget experiment with GLM-4.6V on AndroidControl, a 128-token budget worked best, adding about 1.23 points of action accuracy over no reasoning. Larger budgets did not help further and generally reduced accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China78 works · 33,365 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 worksHuman-Computer Interaction · 160 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Li Zhang, Longxi Gao and Mengwei Xu. 2026. Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.392

@inproceedings{zhang2026does,
  title = {{Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study}},
  author = {Li Zhang, Longxi Gao and Mengwei Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.392},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.392}
}

References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. GLM-V Team. 2026. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006.
  2. Anthropic. 2024. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https://www.anthropic.com/news/ 3-5-models-and-computer-use. ↗
  3. Anthropic. 2025a. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/ claude-3-7-sonnet. ↗
  4. Anthropic. 2025b. Claude’s extended thinking. https://www.anthropic.com/ research/visible-extended-thinking. ↗
  5. Apple. 2024. Siri - apple. https://www.apple.com/siri/. ↗
  6. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. arXiv prepri…
  7. Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, and 1 others. 2024. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264. — [arXiv:2409.08264](https://arxiv.org/abs/2409.08264) find
  8. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok…
  9. Google. 2024. Google assistant, your own personal google. https://assistant.google.com/. ↗
  10. Google. 2025. Gemini flash thinking -google deepmind. https://deepmind.google/technologies/gemini/ flash-thinking/. ↗
  11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  12. Youngmin Im, Byeongung Jo, Jaeyoung Wi, Seungwoo Baek, Tae Hoon Min, Joo Hyung Lee, Sangeun Oh, Insik Shin, and Sunjae Lee. 2025. Modular and multi-path-aware offline benchmarking for mobile gui agents. arXiv preprint arXiv:2512.12634. — [arXiv:2512.12634](https://arxiv.org/abs/2512.12634) find
  13. Kimi. 2025. Kimi-vl technical report. arXiv preprint arXiv:2504.07491.
  14. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981. — [arXiv:2504.07981](https://arxiv.org/abs/2504.07981) find
  15. Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024a. On the effects of data scale on ui control agents. In Advances in Neural Information Processing Systems, volume 37, pages 92130–92154. Curran Associates, Inc.
  16. Google Deepmind. 2025. Gemini 2.5: Our most intel-Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, ligent ai model — blog.google. https://blog. Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenx-google/technology/google-deepmind/ ing Xu, Xiang Wang, Yi Sun, and 1 others. 2024b. gemini-model-thinking-updates…
  17. Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Gongshen Liu, and Zhuosheng Zhang. 2025. Say one Xiaotian Han, Shengyu Zhang, Hongxia Yang, and thing, do another? diagnosing reasoning-execution Fei Wu. 2025… — [arXiv:2510.02204](https://arxiv.org/abs/2510.02204) find
  18. Longxi Gao, Li Zhang, Pengzhi Gao, Wei Liu, Jian Luan, and Mengwei Xu. 2025. Gui-shift: Enhancing vlm-based gui agents through self-supervised reinforcement learning. arXiv preprint arXiv:2505.12493. — [arXiv:2505.12493](https://arxiv.org/abs/2505.12493) find
  19. Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451. — [arXiv:2406.08451](https://arxiv.org/abs/2406.08451) find
  20. OpenAI. 2024. Openai o1 system https://openai.com/index/ openai-o1-system-card/. card. ↗
  21. OpenAI. 2025. Introducing operator. https://openai.com/index/ introducing-operator/. ↗
  22. Charilaos Pipis, Shivam Garg, Vasilis Kontonis, Vaishnavi Shrivastava, Akshay Krishnamurthy, and Dimitris Papailiopoulos. 2025. Wait, wait, wait... why do reasoning models loop? arXiv preprint arXiv:2512.12895. — [arXiv:2512.12895](https://arxiv.org/abs/2512.12895) find
  23. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, and 1 others. 2024. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573.
  24. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. An-droidinthewild: A large-scale dataset for android device control. In Advances in Neural Information Processing Systems, volume 36, pages 59708–59728. Curran Associates, Inc. find
  25. Simular. 2025. Agent s2. https://www.simular.ai/agent-s2. ↗
  26. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
  27. Bryan Wang, Gang Li, and Yang Li. 2023. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17. find
  28. Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint… — [arXiv:2401.06805](https://arxiv.org/abs/2401.06805) find
  29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi…
  30. Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024a. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, ACM Mobi-Com 2024,… find
  31. Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, and 1 others. 2024b. Autodroid-v2: Boosting slm-based gui agents via code generation. arXiv preprint arXiv:2412.18116. — [arXiv:2412.18116](https://arxiv.org/abs/2412.18116) find
  32. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and 1 others. 2024. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218.
  33. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in…
  34. Mengwei Xu. 2025. Every software as an agent: Blueprint and case study. arXiv preprint arXiv:2502.04747. — [arXiv:2502.04747](https://arxiv.org/abs/2502.04747) find
  35. Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. 2024. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024.
  36. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441.
  37. Z.ai. 2025. zai-org/glm-4.6 — huggingface.co. https://huggingface.co/zai-org/ GLM-4.6. ↗
  38. Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang. 2025a. Api agents vs. gui agents: Divergence and convergence. Preprint, arXiv:2503.11069. — [arXiv:2503.11069](https://arxiv.org/abs/2503.11069) find
  39. Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, and 1 others. 2024a. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279. — [arXiv:2411.18279](https://arxiv.org/abs/2411.18279) find
  40. Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024b. Android in the zoo: Chain-of-action-thought for GUI agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 12016– 12031.… find
  41. Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. 2024c. Llamatouch: A faithful and scalable testbed for mobile ui task automation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–13. find
  42. Li Zhang and Mengwei Xu. 2025. Scaling test-time compute in mobile gui agents with parallel speculative execution. In Proceedings of the 2nd International Workshop on Edge and Mobile Foundation Models, pages 25–30. find
  43. Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, and 1 others. 2025b. Agentcpm-gui: Building mobile-use agents with reinforcement fine-tuning. arXiv preprint arXiv:2506.01391. find
  44. Zhuosheng Zhang and Aston Zhang. 2024. You only look at screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 3132–3149. Association for Computational Linguistics.
  45. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-does-chain-thought-reasoning-help/