Elamur Research⌕
☰
home / papers / 2026 / SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents

SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents

Paper · 2026 · Zonghao Ying, Yangguang Shao, Jianle Gan et al.

Summary

Web agents built on large vision–language models (LVLMs) can browse sites, fill in forms and run multi-step tasks. That autonomy also makes them targets. Earlier security benchmarks usually test one threat at a time, either malicious user requests or hostile page content. The authors present SecureWebArena, a benchmark that covers both. It provides six simulated web environments adapted from WebArena, including shopping, Wikipedia and GitLab. Across these, 330 adversarial scenarios produce 2,970 attack trajectories, grouped into six attack vectors at the user level and the environment level. Agents follow the Set-of-Marks (SoM) paradigm: they read annotated screenshots and element metadata and act on element IDs. The evaluation checks three layers: the agent's reasoning, its action trajectory, and the final task outcome. This shows where in the process a failure occurs, not just whether one did. Tests of 9 LVLMs from three model types found that every model is vulnerable. Specialization brings security trade-offs, and no model type resists all attacks. A small test on live websites suggests the weaknesses carry over beyond simulation.

Key points

  • Attack taxonomy:
    • User-level vectors: Jailbreak and Direct Prompt Injection (DP Injection).
    • Environment-level vectors: Indirect Prompt Injection (IP Injection), Pop-up attack, Distract attack and Ad Injection.
  • Layered metrics: results are reported as RVR, BCR and PDR, which correspond to the reasoning, behavior and outcome layers.
  • Specialized models still fail on visual attacks:
    • The agent-specialized GLM-4.5V followed Pop-ups in 96.67% of cases.
    • The GUI-grounded Aguvis recovered during execution, with IP Injection risk falling from 53.33% to 0%, but its early reasoning was heavily compromised.
  • Different root causes under a Pop-up attack: GPT-5 read the text correctly but talked itself into the risky action. UI-TARS-1.5 simply followed what was visually prominent.
  • Live-site test on Amazon and Wikipedia (user-level attacks only):
    • GPT-5 took unsafe actions in over 60% of cases.
    • GLM-4.5V reached 100% compromise under Jailbreak on Wikipedia.
    • UI-TARS-1.5 fully resisted DP Injection but reached 60% PDR under Jailbreak.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of SydneySydney · Australia45 works · 68,350 citations🎓Beihang UniversityBeijing · China65 works · 8,452 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of Information EngineeringBeijing · China30 works · 1,411 citations🎓Zhejiang University of TechnologyHangzhou · China6 works · 98 citations🎓China University of Petroleum, East ChinaQingdao · China2 works · 3 citations

Also on the byline: Henan University of Science and Technology.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Zonghao Ying, Yangguang Shao, Jianle Gan, Gan Xu, Wenxin Zhang, Quanchen Zou, Junzheng Shi, Zhenfei Yin, Mingchuan Zhang, Aishan Liu and Xianglong Liu. 2026. SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.582

@inproceedings{ying2026securewebarena,
  title = {{SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents}},
  author = {Zonghao Ying, Yangguang Shao, Jianle Gan, Gan Xu, Wenxin Zhang, Quanchen Zou, Junzheng Shi, Zhenfei Yin, Mingchuan Zhang, Aishan Liu and Xianglong Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.582},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.582}
}

References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. 2024. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032. — [arXiv:2407.13032](https://arxiv.org/abs/2407.13032) find
  2. Anthropic. 2025a. Introducing claude 3.7 sonnet and claude code. Accessed: 2025-10-04. find
  3. Anthropic. 2025b. Introducing claude 4. Accessed: 2025-10-04. find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  5. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023a. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114.
  6. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023b. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474.
  7. Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2023. A Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.2…
  8. Sam Johnson, Viet Pham, and Thai Le. 2025. Manipulating llm web agents with indirect prompt injection attack via html accessibility tree. arXiv preprint arXiv:2507.14799. — [arXiv:2507.14799](https://arxiv.org/abs/2507.14799) find
  9. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649.
  10. Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, and 1 others. 2024. Refusal-trained llms are easily jailbroken as browser agents. arXiv preprint arXiv:2410.13886. — [arXiv:2410.13886](https://arxiv.org/abs/2410.13886) find
  11. Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and 1 others. 2024. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p…
  12. Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2024. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703. — [arXiv:2410.06703](https://arxiv.org/abs/2410.06703) find
  13. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191.
  14. Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. 2025. Caution for the environment: Multimodal llm agents are susceptible to environmental distractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  15. Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, and 1 others. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In Proceedings of the 31st ACM SIGKDD Conferenc… find
  16. OpenAI. 2025. Gpt-5 is here. Accessed: 2025-10-04. find
  17. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326.
  18. Haoyi Qiu, Alexander R Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. 2024. Evaluating cultural and social awareness of llm web agents. arXiv preprint arXiv:2410.23252. find
  19. Settaluri Lakshmi Sravanthi, Ankit Mishra, Debjyoti Mondal, Subhadarshi Panda, Rituraj Singh, and Pushpak Bhattacharyya. 2025. From perception to reasoning: Enhancing vision-language models for mobile ui understanding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 252… find
  20. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  21. V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, and 1 others. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multi-modal reasoning with scalable reinforcement learning, 2025. URL https://arxiv.org/abs/2507.01006. ↗
  22. Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stańczak, and Siva Reddy. 2025. Safearena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957. — [arXiv:2503.04957](https://arxiv.org/abs/2503.04957) find
  23. Haowei Wang, Junjie Wang, Xiaojun Jia, Rupeng Zhang, Mingyang Li, Zhe Liu, Yang Liu, and Qing Wang. 2025a. Adinject: Real-world black-box attacks on web agents via advertising delivery. arXiv preprint arXiv:2505.21499. — [arXiv:2505.21499](https://arxiv.org/abs/2505.21499) find
  24. Le Wang, Zonghao Ying, Tianyuan Zhang, Siyuan Liang, Shengshan Hu, Mingchuan Zhang, Aishan Liu, and Xianglong Liu. 2025b. Manipulating multimodal agents via cross-modal prompt injection. arXiv preprint arXiv:2504.14348. — [arXiv:2504.14348](https://arxiv.org/abs/2504.14348) find
  25. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  26. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  27. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  28. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. — [arXiv:2412.04454](https://arxiv.org/abs/2412.04454) find
  29. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. find
  30. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  31. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463.
  32. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471.
  33. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
  34. Uses persuasive language to elicit unsafe behavior. Injects modals to hijack navigation. Alters layout to confuse visual saliency. Mimics UI appearance to trigger misclicks. Embeds prompt-like text into interface elements. find
  35. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2024. Attacking vision-language computer agents via pop-ups. arXiv preprint arXiv:2411.02391.
  36. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.
  37. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ying-2026-securewebarena-holistic-security-evaluation-benchmark/