Elamur Research⌕
☰
home / papers / 2026 / Coding Agents with Multimodal Browsing are Generalist Problem Solvers

Coding Agents with Multimodal Browsing are Generalist Problem Solvers

Paper · 2026 · Aditya Bharat Soni, Boxuan Li, Xingyao Wang et al.

Summary

Strong LLM agents for software engineering, deep research and web navigation are usually specialists. Each is built around tools and workflows tuned for a narrow set of benchmarks, so it breaks down outside its home domain. The authors ask whether a single agent with a small set of general-purpose tools can match these specialists everywhere.

They extend the OpenHands coding framework (v0.28.1) into OpenHands-Versa, adding:

  • visual browsing through Set-of-Marks screenshots combined with the AXTree,
  • a condenser that keeps only recent browser observations,
  • web search through an API instead of CAPTCHA-prone browser searches,
  • Markdown-based conversion for viewing multimodal files,
  • a periodic planning prompt.

Evaluated on SWE-Bench Multimodal, GAIA and The Agent Company, this one agent outperforms or rivals the best published specialist systems on all three. The authors also report that specialist multi-agent systems fail to transfer beyond their target tasks. The results suggest that domain-specific engineering offers little practical advantage here, and the code is released as an open-source baseline for work on generalist agents.

Key points

  • Toolset: bash, Jupyter IPython, a Playwright/BrowserGym browser with Set-of-Marks visual input, a search API (Tavily by default, with Exa and Brave also supported) and a multimodal file viewer. All of these run in a single-agent, sandboxed setup.
  • Headline gains: the improvement over prior published results is 9.1 points on The Agent Company. On the other two benchmarks the gains are 9.1 and 1.3 points, but the abstract and the introduction assign these two figures to SWE-Bench Multimodal and GAIA in opposite order.
  • GAIA ablations: removing a single tool hurts performance. The drop is 17.6% without search, 12.7% without the multimodal file viewer and 6.1% without the browser enhancements.
  • Search APIs: Tavily performed best, plausibly because it returns an LLM-synthesized answer that reduces hallucinations from individual page snippets.
  • Limitations:
    • The system depends on closed-source LLMs.
    • Results come from single runs, with no significance testing.
    • Video handling is weak.
    • Planning is simplistic.
    • Ablations were run on GAIA only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksCode & programming languages · 129 works

Cite

Cite as Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen and Graham Neubig. 2026. Coding Agents with Multimodal Browsing are Generalist Problem Solvers. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.318

@inproceedings{soni2026coding,
  title = {{Coding Agents with Multimodal Browsing are Generalist Problem Solvers}},
  author = {Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen and Graham Neubig},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.318},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.318}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude code. https://www.claude.com/product/claude-code. Accessed: 2025-10-01. ↗
  2. Anthropic(2025). The "think" tool: Enabling claude to stop and think in complex tool use situations. A new tool that improves Claude’s complex problem-solving performance. find
  3. Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, and 1 others. 2024a. Tapeagents: a holistic framework for agent development and optimization. arXiv preprint arXiv:2412.08445. find
  4. Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, Soham Parikh, Mitul Tiwari, and Quaizar Vohra. 2024b. Tapeagents: a holistic framework for agent development and optimization. Preprin… find
  5. Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salak… find
  6. Dunn(2025). Agentless-lite. find
  7. Exa. 2023. Exa search api. find
  8. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, and 1 others. 2024. Magentic-one: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468. find
  9. Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Ping Luo, and Guohao Li. 2025. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. find
  10. Brave Software Inc. 2023. Brave search api. find
  11. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  12. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649.
  13. ManusAI. 2025. Manusai. https://manus.im/. Accessed: 2025-10-01. ↗
  14. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations.
  15. Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde-McLaughlin, and 1 others. 2025. Magentic-ui: Towards human-in-the-loop agentic systems. arXiv preprint arXiv:2507.22358. — [arXiv:2507.22358](https://arxiv.org/abs/2507.22358) find
  16. Magnus Müller and Gregor Žunič. 2024. Browser use: Enable ai to control your browser. find
  17. OpenAI. 2025a. Introducing deep research. https://openai.com/index/ introducing-deep-research/. Accessed: 2025-10-01. ↗
  18. OpenAI. 2025b. Introducing operator. https://openai.com/index/introducing-operator/. Accessed: 2025-10-01. ↗
  19. OpenAI. 2025c. Openai codex. https://openai.com/codex/. Accessed: 2025-10-01. ↗
  20. Perplexity AI. 2025. Perplexity comet. https://www.perplexity.ai/comet. Accessed: 2025-10-01. ↗
  21. Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier. 2025. Open-source deepresearch – freeing our search agents. find
  22. Tavily-AI. 2024. Tavily search api. find
  23. Luis von Ahn, Manuel Blum, Nicholas J. Hopper, and John Langford. 2003. Captcha: Using hard ai problems for security. In Advances in Cryptology -EUROCRYPT 2003, International Conference on the Theory and Applications of Cryptographic Techniques, Warsaw, Poland, May 4-8, 2003, Proceedings, volume 265… find
  24. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others. 2024a. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations.
  25. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2024b. Agent workflow memory. arXiv preprint arXiv:2409.07429.
  26. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint. find
  27. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in…
  28. Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, and 2 others. 2024. Theagentcompany: Benchmarking llm ag… find
  29. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. Preprint, arXiv:2310.11441. find
  30. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024a. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  31. John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. 2025. SWE-bench multimodal: Do AI systems generalize to visual software domains? In The Thirteenth Internationa… find
  32. Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. 2024b. Agentoccam: A simple yet strong baseline for llm-based web agents. arXiv preprint arXiv:2410.13825. — [arXiv:2410.13825](https://arxiv.org/abs/2410.13825) find
  33. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-swe-bench: A multilingual benchmark for issue resolving… — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) find
  34. Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Leandro Z Agudelo, Peter Qian, and Tianlong Chen. 2025. Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms. arXiv preprint arXiv:2502.07942. — [arXiv:2502.07942](https://arxiv.org/abs/2502.07942) find
  35. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592–1604. find
  36. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soni-2026-coding-agents-multimodal-browsing-are/