Elamur Research⌕
☰
home / papers / 2026 / WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Paper · 2026 · Chenxu Liu, Yingjie Fu, Wei Yang et al.

Summary

WebCoderBench addresses the difficulty of benchmarking LLMs that turn user requests into web apps. Existing benchmarks fall short in three ways. Synthetic requirement sets do not reflect how real users phrase requests. Reference-based setups rely on fixed screenshots or test cases. Arena-style rankings depend on human votes and give no fine-grained diagnosis. The authors sampled real requests from an industrial partner's online LLM service and turned them into 1,572 single-turn requirements. Each requirement comes with a checklist for functionality, visual design and content, drafted by LLMs and verified by experts. Evaluation is fully automatic and needs no reference implementation. It uses 24 metrics grouped into 9 perspectives, mixing rule-based checks with LLM-as-a-judge scoring. Metric weights come from a survey of user preferences, so the combined score is both interpretable and aligned with what users value. Across 12 LLMs and 2 LLM-based agents, no system wins on every metric. The gap between open-source and closed-source models also appears to be shrinking. The benchmark therefore shows developers where specific models need improvement.

Key points

  • Dataset construction: 5,000 requests were sampled from one week of anonymized traffic and narrowed to 1,572.
    • Multi-turn dialogues were merged into single-turn requirements, with help from Gemini-2.5-pro.
    • Three annotators filtered out items, and final labels were set by majority vote.
    • Duplicates were removed with MinHash and MiniLM embeddings.
  • Dataset composition: the requirements span 20 app categories.
    • By input type, 1,413 are text-only, 123 include images and 36 include URLs.
    • By clarity, 78 are vague, 730 intermediate and 764 clear.
    • They are also grouped into five complexity levels.
  • Checklist generation: GPT-5-Chat, Gemini-2.5-pro and Doubao-Seed-1.6 drafted the reference checklists, which human experts merged and validated.
  • Persona-based weights: preference weights differ by user persona.
    • Designers give visual quality about 20.8% of the weight.
    • Across all respondents, functional alignment ranks highest at 13.8%.
  • Scope and limitations: the benchmark covers only front-end apps written in native HTML.
    • For legal reasons, the data and metric code are not released.
    • Results are published on a Hugging Face leaderboard (CompileError/WebCoderBench).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓The University of Texas at DallasRichardson · United States61 works · 5,614 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang and Tao Xie. 2026. WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.535

@inproceedings{liu2026webcoderbench,
  title = {{WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics}},
  author = {Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang and Tao Xie},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.535},
  url = {https://doi.org/10.18653/v1/2026.acl-long.535}
}

References

51 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luai Al Shalabi, Zyad Shaaban, and Basel Kasasbeh. 2006. Data mining: A preprocessing engine. Journal of Computer Science. find
  2. Anthropic. 2025a. Claude Code. find
  3. Anthropic. 2025b. Overview \| Claude. find
  4. Anysphere. 2025. Cursor. find
  5. Design Arena. 2025. Design Arena: World’s largest crowdsourced benchmark for AI-generated design. find
  6. Tony Beltramelli. 2018. Pix2Code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. find
  7. Tanvir Bhathal and Asanshay Gupta. 2025. Websight: A vision-first architecture for robust web agents. arXiv preprint arXiv:2508.16987. — [arXiv:2508.16987](https://arxiv.org/abs/2508.16987) find
  8. Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences. find
  9. ByteDance. 2025a. Doubao. find
  10. ByteDance. 2025b. Doubao Coding. find
  11. ByteDance. 2025c. TRAE - collaborate with intelligence. find
  12. Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung, and Golnoosh Farnadi. 2025. Neither valid nor reliable? Investigating the use of LLMs as judges. arXiv preprint arXiv:2508.18076. — [arXiv:2508.18076](https://arxiv.org/abs/2508.18076) find
  13. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, and 1 others. 2025. MiniMax-M1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585.
  14. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, and 1 others. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conferen…
  15. World Wide Web Consortium. 2025. W3C standards and drafts. find
  16. ESLint. 2025. ESLint: Find and fix problems in your JavaScript code. find
  17. Google. 2025. Introduction to Lighthouse. find
  18. Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Bohua Chen, Yi Su, Dongping Chen, Siyuan Wu, Xing Zhou, and 1 others. 2025. WebCode2M: A real-world dataset for code generation from webpage designs. In Proceedings of the ACM on Web Conference. find
  19. Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Shaoling Dong, Xing Zhou, and Wenbin Jiang. 2024. Vision2UI: A real-world dataset with layout for code generation from UI designs. arXiv preprint arXiv:2404.06369. — [arXiv:2404.06369](https://arxiv.org/abs/2404.06369) find
  20. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  21. Sebastian Hartwig, Dominik Engel, Leon Sick, Hannah Kniesel, Tristan Payer, Poonam Poonam, Michael Glockler, Alex Bauerle, and Timo Ropinski. 2025. A survey on quality metrics for text-to-image generation. IEEE Transactions on Visualization and Computer Graphics. 11641 10 find
  22. HTML Hint. 2025. HTML Hint: The static code analysis tool you need for your HTML. find
  23. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186.
  24. Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2025. Preference leakage: A contamination problem in LLM-as-a-judge. arXiv preprint arXiv:2502.01534. — [arXiv:2502.01534](https://arxiv.org/abs/2502.01534) find
  25. OpenAI. 2025. Introducing GPT-5 for developers. find
  26. Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, and Jian Kang. 2025. Analyzing uncertainty of LLM-as-a-judge: Interval evaluations with conformal prediction. arXiv preprint arXiv:2509.18658. find
  27. Stylelint. 2025. Stylelint: A mighty CSS linter that helps you avoid errors and enforce conventions. find
  28. Haoyu Sun, Huichen Will Wang, Jiawei Gu, Linjie Li, and Yu Cheng. 2025. FullFront: Benchmarking MLLMs across the full front-end engineering workflow. arXiv preprint arXiv:2505.17399. — [arXiv:2505.17399](https://arxiv.org/abs/2505.17399) find
  29. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and 1 others. 2023. StarCoder: May the source be with you! arXiv preprint arXiv:2305.06161. find
  30. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  31. Ryan Li, Yanzhe Zhang, and Diyi Yang. 2024. Sketch2Code: Evaluating vision-language models for interactive web design prototyping. arXiv preprint arXiv:2410.16232. find
  32. Tung D Vu, Chung Hoang, and Truong-Son Hy. 2025. Multimodal graph representation learning for website generation based on visual sketch. arXiv preprint arXiv:2504.18729. — [arXiv:2504.18729](https://arxiv.org/abs/2504.18729) find
  33. Zhiyu Lin, Zhengda Zhou, Zhiyuan Zhao, Tianrui Wan, Yilun Ma, Junyu Gao, and Xuelong Li. 2025. We-bUIBench: A comprehensive benchmark for evaluating multimodal large language models in webUI-to-code. arXiv preprint arXiv:2506.07818. — [arXiv:2506.07818](https://arxiv.org/abs/2506.07818) find
  34. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems. find
  35. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. find
  36. Windsurf. 2025. Windsurf - the best AI for coding. find
  37. LMArena.ai. 2025. WebDev Arena. find
  38. Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2025. WebGen-Bench: Evaluating LLMs on generating interactive and functional websites from scratch. arXiv preprint arXiv:2505.03733. — [arXiv:2505.03733](https://arxiv.org/abs/2505.03733) find
  39. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics. find
  40. Manus. 2025. Manus: Hands on AI. find
  41. Mumtaz Ali Memon, Hiram Ting, Jun-Hwa Cheah, Ramayah Thurasamy, Francis Chuah, and Tat Huei Cham. 2020. Sample size for survey research: Review and recommendations. Journal of Applied Structural Equation Modeling. find
  42. MiniMax. 2025a. MiniMax agent. find
  43. MiniMax. 2025b. MiniMax M2 & agent: Ingenious in simplicity. find
  44. Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R Lyu. 2025. DesignBench: A comprehensive benchmark for MLLM-based front-end code generation. arXiv preprint arXiv:2506.06251. — [arXiv:2506.06251](https://arxiv.org/abs/2506.06251) find
  45. Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng. 2025. Web-Bench: A LLM code benchmark based on web standards and frameworks. arXiv preprint arXiv:2505.07473. find
  46. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  47. H Peyton Young. 1974. An axiomatization of Borda’s rule. Journal of Economic Theory. find
  48. Sukmin Yun, Rusiru Thushara, Mohammad Bhat, Yongxin Wang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, and 1 others. 2024. Web2Code: A large-scale webpage-to-code dataset and evaluation framework for multimodal LLMs. Advances in Neural Information Processing Systems. find
  49. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. GLM-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471. 11642 11
  50. Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Changzhi Zhou, Ken Deng, Dengpeng Wu, Guanhua Huang, Kejiao Li, and 1 others. 2025. ArtifactsBench: Bridging the visual-interactive gap in LLM code generation evaluation. arXiv preprint arXiv:2507.04952. — [arXiv:2507.04952](https://arxiv.org/abs/2507.04952) find
  51. Hongda Zhu, Yiwen Zhang, Bing Zhao, Jingzhe Ding, Siyao Liu, Tong Liu, Dandan Wang, Yanan Liu, and Zhaojian Li. 2025. FrontendBench: A benchmark for evaluating LLMs on front-end development via automatic evaluation. arXiv preprint arXiv:2506.13832. 11643 12 — [arXiv:2506.13832](https://arxiv.org/abs/2506.13832) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-webcoderbench-benchmarking-web-application-generation/