Elamur Research⌕
☰
home / papers / 2025 / Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks

Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks

Paper · 2025 · Kai Xu, YiWei Mao, XinYi Guan et al.

Summary

Older code benchmarks such as HumanEval and MBPP are close to saturated, and software-engineering suites like SWE-Bench are heading the same way. Their tasks are also isolated, unlike real development, where features build on each other. Web-Bench, from ByteDance, addresses both problems with 50 web projects. Each project is a chain of 20 dependent tasks, so every step extends the code produced in the previous one. Experienced engineers designed the projects to span core Web Standards (DOM, CSS layout, SVG, Canvas, WebGL, ECMAScript, TypeScript) and popular Web Frameworks (UI, state management, CSS tooling, build, fullstack, ORM). They also calibrated the tasks and tests against model runs. Playwright end-to-end tests check every task. A reference agent, Web-Agent, assembles the prompts, calls the model and extracts the generated files. The authors evaluated about 40 models. The strongest, Claude 3.7 Sonnet with thinking, solves only about a quarter of tasks on the first try. Web-Bench therefore remains a demanding target, and the authors argue that LLMs need targeted improvement on both standards and frameworks.

Key points

  • Dataset scale: 50 projects × 20 sequential tasks. Projects average 3.6 E2E testcases per task, 72.4 per project, 23.4 solution files and roughly 1,950 lines of code. A senior engineer needs an estimated 4–8 hours per project.
  • Metrics: Pass@1 counts the tasks solved before the first failure. Pass@2 allows one retry that includes the error context, which the authors consider closer to how humans work.
  • Top result: claude-3-7-sonnet-20250219-thinking reached 25.11% Pass@1 and 35.33% Pass@2. For comparison, SOTA on SWE-Bench is 65.4% (Verified) and 33.8% (Full).
  • Other models: gpt-4.1 scored 21.09% Pass@1 and the best open model, deepseek-v3-0324, scored 17.07%. Several smaller models stayed below 5%.
  • Availability: the code and evaluation tool are on GitHub (bytedance/web-bench), and the data is on Hugging Face (bytedance-research/Web-Bench). The authors plan a Lite version to reduce evaluation cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Kai Xu, YiWei Mao, XinYi Guan and ZiLong Feng. 2025. Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks. arXiv:2505.07473.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.07473 · arXiv 2505.07473

@misc{xu2025web,
  title = {{Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks}},
  author = {Kai Xu, YiWei Mao, XinYi Guan and ZiLong Feng},
  year = {2025},
  doi = {10.48550/arxiv.2505.07473},
  eprint = {2505.07473},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.07473}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents, 2024b. URL https://arxiv.org/abs/2407.16741, 2(4):9, 2024. \#\# 7. Limitation Web 2.0 has brought about the rapid development of the Web development ecosystem. Dating back to 2005… ↗
  2. [4] Tommaso Calò and Luigi De Russis. Leveraging large language models for end-user website generation. In International Symposium on End User Development, pages 52–61. Springer, 2023. find
  3. [5] Marcus Tantakoun, Xiaodan Zhu, and Christian Muise. Llms as planning modelers: A survey for leveraging large language models to construct automated planning models. arXiv preprint arXiv:2503.18971, 2025. — [arXiv:2503.18971](https://arxiv.org/abs/2503.18971) find
  4. This paper mainly studies the accuracy of code generation for LLMs. We have also received feedback from users on multiple occasions expressing interest in evaluating the aesthetics and code quality of the generated pages. Since the evaluation of aesthetics and code quality can largely reuse the exis… — Evaluating Large Language Models Trained on Code
  5. [7] Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906, 2024. \#\# Acknowledgements We thank YuBei Li, Quan Ding, Feng Gu, YuKun Wang, ChaoMo Li, Kai Shen, ShuLin Xin, LinJian Cao, a…
  6. [8] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  7. [9] Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, and Omer Tripp. Qualityflow: An agentic workflow for program synthesis controlled by llm quality checks. arXiv preprint arXiv:2501.17167, 2025. — [arXiv:2501.17167](https://arxiv.org/abs/2501.17167) find
  8. [10] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald References Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: a [1] Arghavan Moradi Dakhel, Vahid Majdinasab, scalable and polyglot approach to benchmarking Amin Ni… find
  9. [12] Niklas Muennighoff, Qian Liu, Armel Zebaze, [21] Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Swayam Singh, Xiangru Tang, Leandro Von Werra, Binyuan Hui, Qicheng Zhang, et al. Devbench: and Shayne Longpre. Oct… find
  10. CoRR, 2024. Workshop on Instruction Tuning and Instruction [22] Carlos E Jimenez, John Yang, Alexander Wettig, Following, 2023. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models [13] Yujia Li, David Choi, Junyoung Chung, Nate resolve real-world github issues?…
  11. Springer, 2014. [15] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau [25] Sanchit Aggarwal et al. Modern web-development using reactjs. International Journal of Recent Re-Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-search Aspects, 5(1):133–137, 2018. 1000:… find
  12. PMLR, 2023. Journal of Computer Sciences Institute, 20:183– 188, 2021. [16] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, [27] John Yang, Carlos Jimenez, Alexander Wettig, Qianxiang Wang, and Tao Xie. Codereval: A Kilian Lieret, Shunyu Yao, Karthik Narasimhan… find
  13. [17] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, [28] Chao Lei, Yanchuan Chang, Nir Lipovetzky, and Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Krista A Ehinger. Planning-driven programming: Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil A large language model programming workflow. Paul, et al. Bi… find
  14. [20] Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2025-web-bench-llm-code-benchmark/