Elamur Research
☰
home / papers / 2024 / GPT-4V(ision) is a Generalist Web Agent, if Grounded

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2401.01614

Summary

The paper asks whether large multimodal models can serve as generalist web agents, meaning systems that carry out natural-language instructions on arbitrary real websites. The authors argue that rendered screenshots carry information more compactly than raw HTML and include content, such as embedded images, that HTML misses. They build SeeAct, which works in two stages. First, GPT-4V looks at a screenshot and writes a textual description of the next step (action generation). Second, that description is mapped to a concrete HTML element and operation (element grounding). The authors compare three grounding schemes: predicting element attributes, choosing among textual HTML candidates, and set-of-mark-style image labels.

SeeAct is evaluated on Mind2Web in the usual offline setting with cached pages. The authors also built a new tool for running agents on live websites. When grounding is done by hand, GPT-4V completes about half of the live tasks, far ahead of text-only and fine-tuned baselines. Automatic grounding, however, still trails this oracle by a wide margin. The work matters because it identifies grounding, not planning, as the main obstacle for vision-based web agents.

Key points

  • With oracle grounding, SeeAct on GPT-4V completes 51.1% of live-website tasks, compared with 13.3% for GPT-4 and 8.9% for FLAN-T5. Other baselines include BLIP-2, LLaVA-1.5 and CogAgent.
  • In one example page, 423 HTML elements take 186,490 GPT-2 tokens, while the same screenshot needs only 1,445 GPT-4V visual tokens.
  • The best grounding method combines HTML text with visuals. It outperforms image annotation (set-of-mark prompting) by up to roughly 10%, a method that caused heavy hallucination on webpages. Even so, it remains about 20–30% below oracle grounding.
  • In-context learning with large models generalizes better to unseen websites. Supervised fine-tuning still wins on websites seen during training.
  • Online and offline results diverge noticeably because a task often admits several valid plans. The authors therefore treat online evaluation as the more faithful measure.
  • Code, data and evaluation tools are released on GitHub (OSU-NLP-Group/SeeAct) under an OPEN-RAIL license. During live runs, every action was manually checked for safety before execution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

121 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.01614 · arXiv 2401.01614

@misc{zheng2024gpt,
  title = {{GPT-4V(ision) is a Generalist Web Agent, if Grounded}},
  author = {Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu},
  year = {2024},
  doi = {10.48550/arxiv.2401.01614},
  eprint = {2401.01614},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.01614}
}

References

51 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akter, S. N., Yu, Z., Muhamed, A., Ou, T., Bauerle, A., Cabrera, Á. A., Dholakia, K., Xiong, C., and Neubig, G. An in-depth look at gemini’s language abilities. 2023. URL https://api.semanticscholar. org/CorpusID:266359502. ↗
  2. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira,… find our card
  3. Anil, G. T. G. R., Borgeaud, S., and et al., Y. W. Gemini: A family of highly capable multimodal models. 2023. URL https://api.semanticscholar. org/CorpusID:266361876. ↗ our card
  4. Chandu, K. R., Bisk, Y., and Black, A. W. Grounding ‘grounding’in nlp. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4283– 4305, 2021. find
  5. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal llm’s referential dialogue magic. ArXiv, abs/2306.15195, 2023. URL https://api.semanticscholar. org/CorpusID:259262082. ↗ our card
  6. Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. 2024. URL https://api.semanticscholar. org/CorpusID:267069082. ↗ our card
  7. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu, A. W., Zhao, V., Huang, Y., Dai, A. M., Yu, H., Petrov, S., hsin Chi, E. H., Dean… find
  8. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023. find our card
  9. Furuta, H., Nachum, O., Lee, K.-H., Matsuo, Y., Gu, S. S., and Gur, I. Multimodal web navigation with instruction-finetuned foundation models. ArXiv, abs/2305.11854, 2023. URL https://api.semanticscholar. org/CorpusID:258823350. — [arXiv:2305.11854](https://arxiv.org/abs/2305.11854) ↗
  10. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 – 414, 2016. URL https://api.semanticscholar. org/CorpusID:8081284. ↗ our card
  11. Gu, Y., Deng, X., and Su, Y. Don’t generate, discriminate: A proposal for grounding language models to real-world environments. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.… find
  12. Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: An advanced diagnostic suite for entangled language hallucination&visual illusion in large vision-language models. 2023. URL https://api.semanticscholar. org/Cor… ↗
  13. Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., and Faust, A. Understanding html with large language models. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https: //api.semanticscholar.org/CorpusID: 252780086. find
  14. Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with planning, long context understanding, and program synthesis. ArXiv, abs/2307.12856, 2023. URL https://api.semanticscholar. org/CorpusID:260126067. ↗
  15. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. X., and Steinhardt, J. Measuring massive multi-task language understanding. ArXiv, abs/2009.03300, 2020. URL https://api.semanticscholar. org/CorpusID:221516475. ↗ our card
  16. Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., and Tang, J. Cogagent: A visual language model for gui agents. 2023. URL https://api.semanticscholar. org/CorpusID:266210390. ↗ our card
  17. Kazemzadeh, S., Ordonez, V., andre Matten, M., and Berg, T. L. Referitgame: Referring to objects in photographs of natural scenes. In Conference on Empirical Methods in Natural Language Processing, 2014. URL https://api.semanticscholar. org/CorpusID:6308361. ↗
  18. Kim, G., Baldi, P., and McAleer, S. M. Language models can solve computer tasks. ArXiv, abs/2303.17491, 2023. URL https://api.semanticscholar. org/CorpusID:257834038. ↗ our card
  19. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. 2024. URL https://api.semanticscholar. org/CorpusID:267199749. ↗ our card
  20. Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisenschlos, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K. Pix2struct: Screenshot parsing as pretraining for visual language understanding. ArXiv, abs/2210.03347, 2022. URL https: //api.semanticscholar.org/CorpusID: 252762394. find
  21. Li, J., Li, D., Savarese, S., and Hoi, S. C. H. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023. URL https: //api.semanticscholar.org/CorpusID: 256390509. find our card
  22. Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. URL https://api. semanticscholar.org/CorpusID:14113767. find our card
  23. Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https: //arxiv.org/abs/1802.08802. find
  24. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, 2023a. find our card
  25. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. ArXiv, abs/2304.08485, 2023b. URL https://api.semanticscholar. org/CorpusID:258179774. ↗ our card
  26. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023c. find
  27. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating llms as agents. ArXiv, abs/2308.03688, 2023d. URL https:… find our card
  28. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv, abs/2209.09513, 2022. URL https://api.semanticscholar. org/CorpusID:252383606. — [arXiv:2209.09513](https://arxiv.org/abs/2209.09513) ↗
  29. Mazumder, S. and Riva, O. Flin: A flexible natural language interface for web navigation. ArXiv, abs/2010.12844, 2020. URL https://api.semanticscholar. org/CorpusID:225067907. ↗
  30. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api.semanticscholar. org/CorpusID:257532815. ↗ our card
  31. Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. URL https://api.semanticscholar. org/CorpusID:259262263. ↗ our card
  32. Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74 – 93, 2015. URL https://api. semanticscholar.org/CorpusI… find
  33. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021. find our card
  34. Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., and Bhattacharyya, P. Scienceqa: a novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23:289 – 301, 2022. URL https://api.semanticscholar. org/CorpusID:250729995. ↗
  35. Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. ArXiv, abs/2306.00245, 2023. URL https://api.semanticscholar. org/CorpusID:258999511. — [arXiv:2306.00245](https://arxiv.org/abs/2306.00245) ↗
  36. Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can clip benefit vision-and-language tasks? arXiv preprint arXiv:2107.06383, 2021. — [arXiv:2107.06383](https://arxiv.org/abs/2107.06383) find
  37. Shi, T., Karpathy, A., Fan, L. J., Hernández, J. Z., and Liang, P. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, 2017. URL https://api. semanticscholar.org/CorpusID:34953552. find
  38. Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about a red circle? visual prompt engineering for vlms. ArXiv, abs/2304.06712, 2023. URL https://api.semanticscholar. org/CorpusID:258108138. — [arXiv:2304.06712](https://arxiv.org/abs/2304.06712) ↗
  39. Sridhar, A., Lo, R., Xu, F. F., Zhu, H., and Zhou, S. Hierarchical prompting assists large language model on web navigation. ArXiv, abs/2305.14257, 2023. URL https://api.semanticscholar. org/CorpusID:258841249. ↗
  40. Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao, J., Liu, Z., and Wang, L. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. 2023. URL https://api.semanticscholar. org/CorpusID:265149992. ↗
  41. Yang, J., Zhang, H., Li, F., Zou, X., yue Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. ArXiv, abs/2310.11441, 2023a. URL https://api.semanticscholar. org/CorpusID:264172201. ↗
  42. Yang, L., Wang, Y., Li, X., Wang, X., and Yang, J. Fine-grained visual prompting. ArXiv, abs/2306.04356, 2023b. URL https://api.semanticscholar. org/CorpusID:259096008. — [arXiv:2306.04356](https://arxiv.org/abs/2306.04356) ↗
  43. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of lmms: Preliminary explorations with gpt-4v(ision). ArXiv, abs/2309.17421, 2023c. URL https://api.semanticscholar. org/CorpusID:263310951. ↗ our card
  44. Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. ArXiv, abs/2207.01206, 2022. URL https://api.semanticscholar. org/CorpusID:250264533. ↗ our card
  45. You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F., and Yang, Y. Ferret: Refer and ground anything anywhere at any granularity. ArXiv, abs/2310.07704, 2023. URL https://api.semanticscholar. org/CorpusID:263834718. — [arXiv:2310.07704](https://arxiv.org/abs/2310.07704) ↗
  46. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark f… find our card
  47. Zhang, X., Lu, Y., Wang, W., Yan, A., Yan, J., Qin, L., Wang, H., Yan, X., Wang, W. Y., and Petzold, L. R. Gpt-4v(ision) as a generalist evaluator for vision-language tasks. ArXiv, abs/2311.01361, 2023. URL https://api.semanticscholar. org/CorpusID:264935635. — [arXiv:2311.01361](https://arxiv.org/abs/2311.01361) ↗
  48. Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual grounding in multi-modal llms. ArXiv, abs/2307.08581, 2023. URL https://api.semanticscholar. org/CorpusID:259937702. — [arXiv:2307.08581](https://arxiv.org/abs/2307.08581) ↗
  49. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A. S. S., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. ArXiv, abs/2304.06364, 2023. URL https://api.semanticscholar. org/CorpusID:258108259. ↗ our card
  50. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5122–5130, 2017. URL https://api.semanticscholar. org/CorpusID:5636055. ↗
  51. Table of Content: 1. Appendix A: Offline Experiments Method Details 2. Appendix B: Markup Type Ablation Study 3. Appendix C: Online Experiment Details 4. Appendix D:Offline Experiment Prompts 5. Appendix E: Error Examples for Grounding via Image Annotation 6. Appendix F: Strong Capability of Plannin… find