Elamur Research⌕
☰
home / papers / 2026 / EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation

EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation

Paper · 2026 · Pei Yang, Wanyi Chen, Ke Wang et al.

Summary

LLMs are increasingly used to write code that sends blockchain transactions. In that setting a single wrong address, unit or decimal can cause irreversible financial loss. Text-similarity metrics such as BLEU and CodeBLEU, and general coding benchmarks such as SWE-bench, do not measure whether such code actually executes correctly. EVM-QuestBench fills this gap. Models receive natural-language instructions and must produce TypeScript scripts that build transaction requests for EVM-compatible chains; the paper's instance uses BNB Smart Chain. The scripts run on a forked chain with snapshot isolation, and validators check transaction receipts and the resulting on-chain state. Each round draws the instruction wording from a template pool and samples fresh numeric values, which discourages memorization. Composite tasks are penalized for using more steps than necessary. The authors tested 20 models over 5 rounds each. Performance varied widely, and doing single actions well did not reliably predict completing multi-step workflows, so the authors recommend reporting the two splits separately. The benchmark is cheap to run and easy to extend.

Key points

  • Task set: 107 tasks in all, together worth up to 10,700 points.
    • 62 Atomic tasks: basic_transactions (40), defi_operations (19), advanced_features (3).
    • 45 Composite tasks, such as approve→swap, needing 2–6 optimal steps (mean 3.27, mostly 3).
  • Low authoring cost: a new atomic task needs only a JSON specification plus a validator, and a new composite task needs only a JSON file. There are 373 LLM-generated instruction templates in total, which were clarity-rated by three models.
  • Stable rankings: across the 5 rounds, model rankings agreed closely (average Spearman's ρ = 0.960 between round pairs, all p < 0.001).
  • Split asymmetry: some models score well on Composite but weakly on Atomic, while others fail Composite workflows despite reasonable Atomic scores.
  • Cost and portability: a full run can take as little as about 334K tokens (around $0.06). The same architecture has already been ported to Solana. Code is on GitHub (OpenEdgeHQ/EVM-quest-bench).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Soochow UniversitySuzhou · China63 works · 4,507 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksCode & programming languages · 129 works

Cite

Cite as Pei Yang, Wanyi Chen, Ke Wang, Lynn Ai, Eric Yang and Tianyu Shi. 2026. EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1642

@inproceedings{yang2026evm,
  title = {{EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation}},
  author = {Pei Yang, Wanyi Chen, Ke Wang, Lynn Ai, Eric Yang and Tianyu Shi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1642},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1642}
}

References

9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. find
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Noam Shazeer, and Malte Taropa. 2023. PaLM 2 technical report. arXiv preprint arXiv:2305.10403.
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and Jared Kaplan. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  5. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of ICLR.
  6. Mohammad Abdullah Matin Khan, M Saiful Bari, Do Xuan Long, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. XCodeEval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of ACL.
  7. Solana Foundation. 2025. Introducing solana bench: How well can LLMs build complex transactions? Solana News. find
  8. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2023. Execution-based evaluation for open-domain code generation. In Findings of EMNLP. find
  9. Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, and Shuiguang Deng. 2024. CodeScope: An execution-based multilingual multitask multidimensional benchmark for evaluating LLMs on code understanding and generation. In Proceedings… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-evm-questbench-execution-grounded-benchmark/