Elamur Research⌕
☰
home / papers / 2026 / How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

Paper · 2026 · Kuai Yu, Naicheng Yu, Han Wang et al.

Summary

The paper asks how purely visual properties of a webpage steer the choices of VLM-based web agents in benign, non-adversarial settings. Earlier work covered mainly textual signals (price, ratings) or attacks. The authors build VAF, a three-stage evaluation pipeline:

  1. Variant generation. They take HTML snapshots from five real websites, pick one target item per page, and change only its CSS. The item's meaning and function stay fixed while its appearance changes.
  2. Browsing. Agents move through the long page in a human-like way. They receive successive viewport screenshots and decide whether to scroll or click.
  3. Scoring. Behaviour on each variant is compared with the original page using Target Click Rate and Target Mention Rate. The mention rate records whether the agent's reasoning refers to the target item.

Background colour contrast, card size, placement and card sharpness shift agent decisions strongly. Typography, text colour and image blur barely matter. Agents resemble humans in reacting to salience and position, but they recover poorly once visual grounding is disrupted. The findings matter for agent robustness and for understanding how interface design could bias automated decisions.

Key points

  • Scale: 8 variant families with 48 variants per target item. The families alter the background and text colours, the typeface and its size, where the item is placed, how large its card is, how sharp it appears, and the sequence in which items are listed.
  • Websites: Amazon (laptops), eBay (headphones), Booking (San Francisco hotels), Expedia (Yellowstone hotels) and NPR (news).
  • Agents evaluated: UI-TARS 7B, OpenAI CUA, Qwen3VL-8B and GLM4.1v-9B.
    • Position variants, especially the sidebar, were the strongest distractors across models.
    • Card blur hurt more than image blur, which suggests agents rely heavily on item text.
  • Example: On Booking, moving the target from first to a middle slot dropped UI-TARS 7B's click rate from 18% to 0% over 50 trials.
  • Code availability: The abstract links a GitHub repository (ASTRAL-Group/WebAgent_Visual_Attribution). However, the ethics statement says the full codebase will only be released later.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States254 works · 56,747 citations🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Kuai Yu, Naicheng Yu, Han Wang, Rui Yang and Huan Zhang. 2026. How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1723

@inproceedings{yu2026how,
  title = {{How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors}},
  author = {Kuai Yu, Naicheng Yu, Han Wang, Rui Yang and Huan Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1723},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1723}
}

References

53 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amine Allouah, Omar Besbes, Josué Figueroa, Yash Kanoria, and Akshit Kumar. 2025. What is your ai agent buying? evaluation, implications, and emerging questions for agentic e-commerce. Evaluation, Implications, and Emerging Questions for Agentic E-Commerce (August 04, 2025). find
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  3. Manuel Cherep, Chengtian Ma, Abigail Xu, Maya Shaked, Pattie Maes, and Nikhil Singh. 2025. A framework for studying ai agent behavior: Evidence from consumer choice experiments. arXiv preprint arXiv:2509.25609. — [arXiv:2509.25609](https://arxiv.org/abs/2509.25609) find
  4. De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F Xu, Siva Reddy, Quentin Cappart, and 1 others. 2024. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467. find
  5. Jeffrey Yang Fan Chiang, Seungjae Lee, Jia-Bin Huang, Furong Huang, and Yizheng Chen. 2025. Why are web ai agents more vulnerable than standalone llms? a security analysis. arXiv preprint arXiv:2502.20383. — [arXiv:2502.20383](https://arxiv.org/abs/2502.20383) find
  6. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. Advances in Neural Information Processing Systems, 37:82895–82920. find
  7. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114.
  8. Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. 2025. Wasp: Benchmarking web agent security against prompt injection attacks. arXiv preprint arXiv:2504.18575. — [arXiv:2504.18575](https://arxiv.org/abs/2504.18575) find
  9. Brian J Fogg, Jonathan Marshall, Othman Laraki, Alex Osipovich, Chris Varma, Nicholas Fang, Jyoti Paul, Akshay Rangnekar, John Shon, Preeti Swani, and 1 others. 2001. What makes web sites credible? a report on a large quantitative study. In Proceedings of the SIGCHI conference on Human factors in co… find
  10. Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854. — [arXiv:2305.11854](https://arxiv.org/abs/2305.11854) find
  11. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243.
  12. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856.
  13. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.
  14. William E Hick. 1952. On the rate of gain of information. Quarterly Journal of experimental psychology, 4(1):11–26. find
  15. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006.
  16. Jiepu Jiang, Daqing He, and James Allan. 2014. Searching, browsing, and clicking in a search session: Changes in user behavior by task and over time. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 607–616. find
  17. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  18. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computatio…
  19. Rachel Krause. 2022. The dos and don’ts of pairing typefaces. Nielsen Norman Group. Accessed 2025-10-08. find
  20. Luis A Leiva, Yunfei Xue, Avya Bansal, Hamed R Tavakoli, Tuðçe Köroðlu, Jingzhou Du, Niraj R Dayama, and Antti Oulasvirta. 2020. Understanding visual saliency in mobile user interfaces. In 22nd International conference on human-computer interaction with mobile devices and services, pages 1–12. find
  21. Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2024. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703. — [arXiv:2410.06703](https://arxiv.org/abs/2410.06703) find
  22. Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2024. Eia: Environmental injection attack on generalist web agents for privacy leakage. arXiv preprint arXiv:2409.11295. — [arXiv:2409.11295](https://arxiv.org/abs/2409.11295) find
  23. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2025. Showui: One vision-language-action model for gui visual agent. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19498– 19508. find
  24. Xing Han Lù, Gaurav Kamath, Marius Mosbach, and Siva Reddy. 2025. Build the web for agents, not agents for the web. arXiv preprint arXiv:2506.10953. — [arXiv:2506.10953](https://arxiv.org/abs/2506.10953) find
  25. Ho Yin Ng, Zeyu He, and 1 others. 2024. What color scheme is more effective in assisting readers to locate information in a color-coded article? In 2024 IEEE Visualization and Visual Analytics (VIS), pages 291– 295. IEEE. find
  26. Jakob Nielsen. 1999. Designing web usability: The practice of simplicity. New riders publishing. find
  27. Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, and 1 others. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In Proceedings of the 31st ACM SIGKDD Conferenc… find
  28. OpenAI. 2024. Openai computer use agent api. https://platform.openai.com/docs. Commercial vision-language agent model; accessed via API. ↗
  29. Gerardo Perez, Galt P Barber, Anna Benet-Pages, Jonathan Casper, Hiram Clawson, Mark Diekhans, Clay Fischer, Jairo Navarro Gonzalez, Angie S Hinrichs, Christopher M Lee, and 1 others. 2025. The ucsc genome browser database: 2025 update. Nucleic acids research, 53(D1):D1243–D1249. find
  30. Kara Pernice. 2018. Banner blindness revisited: Users dodge ads on mobile and desktop. Nielsen Norman Group, pages 1–18. find
  31. Joshua Porter. 2017. The button color a/b test: Red beats green. Retrieved August, 6:2017. find
  32. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326.
  33. Minjie Shen, Yanshu Li, Lulu Chen, and Qikai Yang. 2025. From mind to machine: The rise of manus ai as a fully autonomous digital agent. find
  34. Mads Soegaard. 2020. Visual hierarchy: Organizing content to follow natural eye movement patterns. Interaction Design Foundation [online]. Aarhus: Interaction Design Foundation, 2. find
  35. Hyunjin Song and Norbert Schwarz. 2008. If it’s hard to read, it’s hard to do: Processing fluency affects effort prediction and motivation. Psychological science, 19(10):986–988. find
  36. Han Wang, An Zhang, Nguyen Duy Tai, Jun Sun, Tat-Seng Chua, and 1 others. 2024a. Ali-agent: Assessing llms’ alignment with human values via agent-based evaluation. Advances in Neural Information Processing Systems, 37:99040–99088. find
  37. Jiwang Yin, Xiaodong Qiu, and Ya Wang. 2025. The impact of ai-personalized recommendations on clicking intentions: Evidence from chinese e-commerce. Journal of Theoretical and Applied Electronic Commerce Research, 20(1):21. find
  38. Ruiqi Wang, Yuqi Jia, and Neil Zhenqiang Gong. 2025a. Obliinjection: Order-oblivious prompt injection attack to llm agents with multi-source data. find
  39. Sou Yoshihara, Taiki Fukiage, and Shin’ya Nishida. 2023. Does training with blurred images bring convolutional neural networks closer to humans with respect to robust object recognition and internal representations? Frontiers in Psychology, 14:1047694. find
  40. Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, and 1 others. 2024b. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890. — [arXiv:2411.04890](https://arxiv.org/abs/2411.04890) find
  41. Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, and Neil Zhenqiang Gong. 2025b. Webinject: Prompt injection attack to web agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2010–2030. find
  42. Jen-Her Wu and Yufei Yuan. 2003. Improving searching and reading performance: the effect of highlighting and text color coding. Information & Management, 40(7):617–637. find
  43. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, and 1 others. 2025. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143. — [arXiv:2506.03143](https://arxiv.org/abs/2506.03143) find
  44. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. 2024. Advweb: Controllable black-box attacks on vlm-powered web agents. arXiv preprint arXiv:2410.17401. — [arXiv:2410.17401](https://arxiv.org/abs/2410.17401) find
  45. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  46. Tao Yu, Zhengbo Zhang, Zhiheng Lyu, Junhao Gong, Hongzhu Yi, Xinming Wang, Yuxuan Zhou, Jiabing Yang, Ping Nie, Yan Huang, and 1 others. 2025. Browseragent: Building web agents with human-inspired web browsing actions. find
  47. Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and 1 others. 2024. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:110935–110971. find
  48. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2025. Attacking vision-language computer agents via pop-ups. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8387–8401.
  49. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614.
  50. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.
  51. Jingqi Yang, Zhilong Song, Jiawei Chen, Mingli Song, Sheng Zhou, Xiaogang Ouyang, Chun Chen, Can Wang, and 1 others. 2025b. Gui-robust: A comprehensive dataset for testing gui agent robustness in real-world anomalies. arXiv preprint arXiv:2506.14477. — [arXiv:2506.14477](https://arxiv.org/abs/2506.14477) find
  52. Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. 2024. Agentoccam: A simple yet strong baseline for llm-based web agents. find
  53. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2026-how-do-visual-attributes-influence/