Elamur Research⌕
☰
home / papers / 2026 / LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark

LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark

Paper · 2026 · Guangyi Liu, Pengxiang Zhao, Liang Liu et al.

Summary

Mobile GUI agents built on large language models often fail in rare, app-specific situations that pre-training or fine-tuning cannot fully cover. The authors propose learning from a handful of user-provided demonstrations. Two things were missing: a benchmark for evaluating this ability on phones, and a framework that works without the structured UI metadata that web-oriented methods depend on. They address both. First, they release LearnGUI, which pairs query tasks with similar k-shot support demonstrations. It has an offline part derived from AMEX and an online part built on AndroidWorld. Second, they introduce LearnAct, a vision-only agent with three modules. DemoParser converts demonstrations into semantically described knowledge entries, KnowSeeker retrieves the most relevant entries, and ActExecutor uses them during task execution. Across six backbone models, LearnAct improves both general-purpose and specialized agents, and offline gains carry over to live tasks. The results suggest that using demonstrations well can partly substitute for model scale in practical mobile automation.

Key points

  • LearnGUI contains 2,252 offline and 101 online tasks, with 2,353 instructions across 73 apps and about 13.2 steps per task on average. It provides k=1–3 demonstrations scored for instruction, UI, and action similarity. The offline test split uses 251 tasks from 9 apps that do not appear in training.
  • Offline, Gemini-2.5-Pro improves from 38.5% to 58.9% with LearnAct.
  • Online, UI-TARS-7B-SFT rises from 18.1% to 32.8%, close to GPT-4o's 34.5%. Qwen3-VL-8B-Instruct goes from 47.6% to 56.9%, and Qwen2-VL-7B-Instruct from 9.9% to 21.1%.
  • In the ablation, the full framework reaches 51.7% against a 19.3% baseline. Removing DemoParser drops accuracy to 40.6%, and removing KnowSeeker (using a random same-app demonstration instead) drops it to 41.6%.
  • Weaker models depend more on demonstrations that closely match the target task. Action similarity helps most when UI similarity is low. The stated limitations are that the benchmark is Android-only and that demonstrations always come from the same app as the query task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHuman-Computer Interaction · 141 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Yaozhen Liang, WenHao Wang, Siheng Chen, Zhengxi Lu, Shuai Ren, Hao Wang, Shibo He, Yong Liu and Wenchao Meng. 2026. LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1491

@inproceedings{liu2026learnact,
  title = {{LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark}},
  author = {Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Yaozhen Liang, WenHao Wang, Siheng Chen, Zhengxi Lu, Shuai Ren, Hao Wang, Shibo He, Yong Liu and Wenchao Meng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1491},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1491}
}

References

54 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Developing a computer use model. find
  2. Li, and 45 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.
  3. Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2021. Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interactive visual environments. arXiv preprint arXiv:2104.08560. — [arXiv:2104.08560](https://arxiv.org/abs/2104.08560) find
  4. Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. 2024. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407.17490. find
  5. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, and 1 others. 2024. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317. — [arXiv:2406.11317](https://arxiv.org/abs/2406.11317) find
  6. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935.
  7. Pengzhou Cheng, Zheng Wu, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang, and Gongshen Liu. 2025. Os-kairos: Adaptive interaction for mllm-powered gui agents. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6701–6725. find
  8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  9. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, and 1 others. 2025a. Seed1.5-vl technical report. arXiv preprint arXiv:2505.07062.
  10. Yuan Guo, Tingjia Miao, Zheng Wu, Pengzhou Cheng, Ming Zhou, and Zhuosheng Zhang. 2025b. Atomic-to-compositional generalization for mobile agents with a new benchmark and scheduling system. arXiv preprint arXiv:2506.08972. — [arXiv:2506.08972](https://arxiv.org/abs/2506.08972) find
  11. Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and 1 others. 2021. Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107.13731. — [arXiv:2107.13731](https://arxiv.org/abs/2107.13731) find
  12. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and 1 others. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290.
  13. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya R… — GPT-4o System Card
  14. Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. 2024. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents. arXiv preprint arXiv:2402.03610. — [arXiv:2402.03610](https://arxiv.org/abs/2402.03610) find
  15. Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.03679. find
  16. Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776.
  17. Yinheng Li, Hailey Hultquist, Justin Wagle, and Kazuhito Koishida. 2025. Instruction agent: Enhancing agent with expert demonstration. arXiv preprint arXiv:2509.07098. — [arXiv:2509.07098](https://arxiv.org/abs/2509.07098) find
  18. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. Showui: One vision-language-action model for gui visual agent. arXiv preprint arXiv:2411.17465. — [arXiv:2411.17465](https://arxiv.org/abs/2411.17465) find
  19. Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, and 1 others. 2025. Llm-powered gui agents in phone automation: Surveying progress and prospects. arXiv preprint arXiv:2504.19838. — [arXiv:2504.19838](https://arxiv.org/abs/2504.19838) find
  20. Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, and 1 others. 2026. Memgui-bench: Benchmarking memory of mobile gui agents in dynamic environments. arXiv preprint arXiv:2602.06075. — [arXiv:2602.06075](https://arxiv.org/abs/2602.06075) find
  21. Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Boyu Wu, Yawen Wang, Jun Hu, and Qing Wang. 2024. Vision-driven automated mobile gui testing via multimodal large language model. arXiv preprint arXiv:2407.03037. — [arXiv:2407.03037](https://arxiv.org/abs/2407.03037) find
  22. Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024a. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451. — [arXiv:2406.08451](https://arxiv.org/abs/2406.08451) find
  23. Pawel Pawlowski, Krystian Zawistowski, Wojciech La-pacz, Marcin Skorupa, Adam Wiacek, Sebastien Postansque, and Jakub Hoscilowicz. 2024. Tinyclick: Single-turn agent for empowering gui automation. arXiv preprint arXiv:2410.11871. find
  24. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326.
  25. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, and 1 others. 2024a. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573. find
  26. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2024b. An-droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36. find
  27. Yunpeng Song, Yiheng Bian, Yongtao Tang, and Zhongmin Cai. 2023. Navigating interfaces with ai for enhanced user interaction. arXiv preprint arXiv:2312.11190. — [arXiv:2312.11190](https://arxiv.org/abs/2312.11190) find
  28. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  29. Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. 2022. Ugif: Ui grounded instruction following. arXiv preprint arXiv:2211.07615. — [arXiv:2211.07615](https://arxiv.org/abs/2211.07615) find
  30. Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, and Manuela Veloso. 2025. Adaptagent: Adapting multimodal web agents with few-shot learning from human demonstrations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find
  31. Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024a. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01014. — [arXiv:2406.01014](https://arxiv.org/abs/2406.01014) find
  32. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024b. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203. — [arXiv:2408.00203](https://arxiv.org/abs/2408.00203) find
  33. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024b. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158. — [arXiv:2401.16158](https://arxiv.org/abs/2401.16158) find
  34. Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, and Giovanni Campagna. 2024. Wilbur: Adaptive in-context learning for robust and accurate web agents. arXiv preprint arXiv:2404.05902. — [arXiv:2404.05902](https://arxiv.org/abs/2404.05902) find
  35. Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. 2024c. Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents. arXiv preprint arXiv:2406.08184. — [arXiv:2406.08184](https://arxiv.org/abs/2406.08184) find
  36. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024d. Qwen2-vl: Enhancing vision-language model’s perception of the…
  37. Yuyang Wanyan, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Jiabo Ye, Yutong Kou, Ming Yan, Fei Huang, Xiaoshan Yang, and 1 others. 2025. Look before you leap: A gui-critic-r1 model for pre-operative error diagnosis in gui automation. arXiv preprint arXiv:2506.04614. — [arXiv:2506.04614](https://arxiv.org/abs/2506.04614) find
  38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  39. Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543–557. find
  40. Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. 2023. Droidbot-gpt: Gpt-powered ui automation for android. arXiv preprint arXiv:2304.07061. — [arXiv:2304.07061](https://arxiv.org/abs/2304.07061) find
  41. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and 1 others. 2024. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218.
  42. Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, and 1 others. 2025. Ui-genie: A self-improving approach for iteratively boosting mllm-based mobile gui agents. arXiv preprint arXiv:2505.21496. — [arXiv:2505.21496](https://arxiv.org/abs/2505.21496) find
  43. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. find
  44. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771. find
  45. Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. 2024a. Mobileexperts: A dynamic tool-enabled agent team in mobile devices. arXiv preprint arXiv:2407.03913. — [arXiv:2407.03913](https://arxiv.org/abs/2407.03913) find
  46. Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024b. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713. find
  47. Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. 2024c. Llamatouch: A faithful and scalable testbed for mobile ui automation task evaluation. arXiv preprint arXiv:2404.16054. — [arXiv:2404.16054](https://arxiv.org/abs/2404.16054) find
  48. Zhuosheng Zhang and Aston Zhang. 2023. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309.11436.
  49. Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, Yuehao Huang, Yaxuan Guo, Kexin Zhang, Hao Wang, Liang Liu, and 1 others. 2025. Mas-bench: A unified benchmark for shortcut-augmented hybrid mobile gui agents. arXiv preprint arXiv:2509.06477.
  50. Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. 2023. Synapse: Trajectory-as-exemplar prompting with memory for computer control. arXiv preprint arXiv:2306.07863. — [arXiv:2306.07863](https://arxiv.org/abs/2306.07863) find
  51. Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, and Hongsheng Li. 2026. Ui-mem: Self-evolving experience memory for online reinforcement learning in mobile gui agents. arXiv preprint arXiv:2602.05832. — [arXiv:2602.05832](https://arxiv.org/abs/2602.05832) find
  52. Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. 2024a. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024.
  53. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024b. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. — [arXiv:2412.04454](https://arxiv.org/abs/2412.04454) find
  54. Dataset \\\# Inst. \\\# Apps \\\# Step HL LL GT FS 187 276 16,660 523 30,378 2,504 15,283 2,946 50 Static Datasets 4 4.2 125 4.5 -1 12 6.3 357 6.5 70 7.5 833 4.8 110 12.8 10 - ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ Benchmarks with Dynamic Environments LlamaTouch 496 57 7.0… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-learnact-few-shot-mobile-gui/