Elamur Research⌕
☰
home / papers / 2026 / MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents

MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents

Paper · 2026 · Xiangyu Wu, Yuwei Hu, Tianyu Cui et al.

Summary

Web agents built on multimodal LLMs still stall when a site asks them to solve a visual CAPTCHA, and popular agent benchmarks rarely test this. Earlier CAPTCHA benchmarks are either very small or synthetic and uniform. The authors built MirrorCAPTCHA to reflect what agents actually encounter online.

They took high-traffic desktop sites from the Common Crawl web graph and sent autonomous agents through login and registration flows to trigger CAPTCHAs. The CAPTCHAs they found were grouped into 18 types with K-means. Random walks over a Common Crawl subgraph estimated how often each type appears, and these frequencies weight the evaluation. Puzzles are solved on interactive webpages through clicks, key presses and drags. Results are scored with Weighted Pass Rate and a new graded metric, Completion Degree.

The authors also trained Ovis2-Agent-CAPTCHA-8B on synthetic CAPTCHA data. It beats strong closed-source models, including the runner-up Gemini-2.5-Pro. The benchmark gives a more realistic picture of agent automation ability and can help defenders see which CAPTCHA types are weak.

Key points

  • Website selection: The pipeline started from the top 15,000 nodes of a Common Crawl graph with 156.1M nodes and 2.1B edges.
    • Accessibility was checked with a modified WebVoyager using Claude-4-Sonnet, which left 10,000 sites.
    • Of these, 2,095 sites were found to deploy CAPTCHAs.
  • Puzzle set: The benchmark contains 1,000 curated puzzles across 18 categories, such as Patch Select, OCR Grey, Slide Puzzle, Rotation Match and Type Arithmetic.
    • Most were collected from real sites and annotated by hand; a small share comes from MCA-Bench.
  • Main result: Ovis2-Agent-CAPTCHA-8B was trained on about 370k synthetic samples. Compared with Gemini-2.5-Pro, it scores:
    • +9.4% average Weighted Pass Rate;
    • +2.13% average Completion Degree;
    • +30.66% Weighted Pass Rate on the high-traffic Patch Select category.
  • Error analysis: Failures come mostly from imprecise recognition of object attributes and counts, inaccurate click coordinates, and difficulty with abstract icons.
  • Limited release: The paper, metrics and framework code are public.
    • The dataset and synthesis pipeline are available only under a Data Use Agreement.
    • The model weights and the full 370k training set are not released.
    • Audio and video CAPTCHAs are not yet covered.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Xiangyu Wu, Yuwei Hu, Tianyu Cui, Yueying Tian, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Yang Yang and Jianfeng Lu. 2026. MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1431

@inproceedings{wu2026mirrorcaptcha,
  title = {{MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents}},
  author = {Xiangyu Wu, Yuwei Hu, Tianyu Cui, Yueying Tian, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Yang Yang and Jianfeng Lu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1431},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1431}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. 2025. Agent s: An open agentic framework that uses computers like a human. In ICLR. find
  2. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, and 1 others. 2023. Gemini: A family of highly capable multimodal models. In arXiv.
  3. Anthropic. 2025. Claude api. Anthropic’s Conversational AI API. find
  4. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, and 1 others. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. In arXiv.
  5. Common Crawl. 2007. Common crawl. Common Crawl corpus contains petabytes of data, regularly collected since 2008. find
  6. Zaid Derea, Beiji Zou, Amal A. Al-Shargabi, Alaa Thobhani, and Amr Abdussalam. 2023. Deep learning based CAPTCHA recognition network with grouping strategy. Sensors, 23(23):9487. find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and 1 others. 2024. The llama 3 herd of models. In arXiv.
  8. Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2025. Plan-and-act: Improving planning of agents for long-horizon tasks. In arXiv. find
  9. Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2024. A real-world webagent with planning, long context understanding, and program synthesis. In ICLR.
  10. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, and 1 others. 2021. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748–8763.
  11. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. In ACL, pages 6864–6890.
  12. Chenyang Shao, Xinyuan Hu, Yutang Lin, and Fengli Xu. 2025. Division-of-thoughts: Harnessing hybrid language model synergy for efficient on-device agents. In WWW, pages 1822–1833. find
  13. Yu Hu, Li Chen, and Jun Cheng. 2018. A captcha recognition technology based on deep learning. In ICIEA, pages 617–620. find
  14. Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P. Bigham, and Graham Neubig. 2025. Cowpilot: A framework for autonomous and human-agent collaborative web navigation. In arXiv. find
  15. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In ACL, pages 881–905.
  16. Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. 2024. Autowebglm: A large language model-based web navigating agent. In KDD, pages 5295–5306.
  17. Yujin Shu and Yongjin Xu. 2019. End-to-end captcha recognition using deep cnn-rnn network. In IMCEC, pages 54–58. find
  18. Significant Gravitas. 2023. AutoGPT. find
  19. Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, Yuan Huang, and Ling Chen. 2024. Mmac-copilot: Multi-modal agent collaboration operating system copilot. In arXiv. find
  20. Shengyuan Tang. 2024. Research on captcha recognition technology based on deep learning. Applied and Computational Engineering, 81:41–46. find
  21. Alaa Thobhani, Mingsheng Gao, Ammar Hawbani, Safwan Taher Mohammed Ali, and Amr Abdussalam. 2020. Captcha recognition using deep learning with attached binary images. Electronics, 9(9). find
  22. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. In arXiv.
  23. Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2025. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. In arXiv. find
  24. Zonglin Wu, Yule Xue, Xin Wei, and Yiren Song. 2025. Mca-bench: A multimodal benchmark for evaluating CAPTCHA robustness against vlm-based attacks. In arXiv. find
  25. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, and 1 others. 2024. Agentbench: Evaluating llms as agents. In ICLR.
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and 1 others. 2025. Qwen3 technical report. In arXiv.
  27. Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model. In arXiv. find
  28. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v(ision). In arXiv.
  29. Yaxin Luo, Zhaoyi Li, Jiacheng Liu, Jiacheng Cui, Xiaohan Zhao, and Zhiqiang Shen. 2025. Open captchaworld: A comprehensive web-based platform for testing and benchmarking multimodal LLM agents. In NeurIPS. find
  30. Guixin Ye, Zhanyong Tang, Dingyi Fang, Zhanxing Zhu, Yansong Feng, Pengfei Xu, Xiaojiang Chen, Jungong Han, and Zheng Wang. 2020. Using generative adversarial networks to break and protect text captchas. ACM Transactions on Privacy and Security, 23. find
  31. Magnus Müller and Gregor Žunič. 2024. Browser use: Enable ai to control your browser. find
  32. A OpenAI. 2025. Openai api. Accessed: 2023-12-01. find
  33. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, and 1 others. 2025. Ui-tars: Pioneering automated GUI interaction with native agents. In arXiv.
  34. Comparison with Existing Benchmarks Table 4 presents a detailed comparison of the three most recent CAPTCHA benchmarks. Both Open CaptchaWorld (Luo et al., 2025) and MCA-Bench (Wu et al., 2025) rely on a random distribution of CAPTCHA types, which fails to accurately reflect the real-world frequenci… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-mirrorcaptcha-wild-captcha-wild-distribution/