Elamur Research⌕
☰
home / papers / 2026 / LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking

LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking

Paper · 2026 · Neeva Oza, Ishaan Govil, Parul Gupta et al.

Summary

The paper asks whether large language models can reliably tell if two programs are functionally equivalent, meaning they return identical outputs for every input. This skill matters for checking code rewrites and translations. The authors build CETBench from CodeContests solutions. They sample program pairs (identical, equivalent, correct versus incorrect, or solving different problems) and pass them through random chains of predefined edits. Some edits keep semantics intact (sp) and others break them (np). Open and closed models, including Qwen, Mistral, DSC and GPT-4o, lose substantial accuracy once these fairly simple edits are applied. The drop is especially large when both programs come from the same original solution, and it grows further when the two programs are in different languages. The authors then fine-tune models on a mix of original and perturbed pairs. This raises scores and carries over without retraining to harder problems and to cross-language pairs. Overall, the authors argue that LLMs lean on surface statistics rather than real semantic understanding of code.

Key points

  • Source data: CETBench is derived from DeepMind's CodeContests, which has roughly 10,000 competitive-programming problems, many from Codeforces. It uses Python, Java and C++ solutions, but only the Python programs are perturbed.
  • Transformation design: each pair receives a composed sequence of up to seven transformations, such as variable renaming or if-else block swaps. By contrast, SeqCoBench uses single-step edits, and EquiBench uses one or two transformation types.
  • Distinctive features: negative pairs differ only through subtle meaning-breaking edits, and prompts omit problem descriptions. The authors present it as the first benchmark to include cross-lingual pairs (Python–Java, Python–C++).
  • Transfer results: models fine-tuned only on 900-level Python–Python pairs were tested on difficulty levels 900–2200 and on cross-language pairs. For DSC, cross-language scores rose from about 35 to about 75. The models that started weakest gained the most.
  • Release and limitations: the dataset and code are available at github.com/dair-iitd/CETBench. Acknowledged limitations are binary Yes/No labels, a non-exhaustive transformation set, and unclear generalisation to unseen perturbation types.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓Indian Institute of Technology DelhiNew Delhi · India22 works · 1,036 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksComputer Science Applications · 102 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg and Parag Singla. 2026. LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2070

@inproceedings{oza2026llms,
  title = {{LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking}},
  author = {Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg and Parag Singla},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2070},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2070}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ajmain I Alam, Palash R Roy, Farouq Al-Omari, Chanchal K Roy, Banani Roy, and Kevin A Schneider. 2023. Gptclonebench: A comprehensive benchmark of semantic clones and cross-language clones using gpt-3 model and semanticclonebench. In 2023 IEEE International Conference on Software Maintenance and Evo… find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. ArXiv, abs/2108.07732.
  3. Sahar Badihi, Yi Li, and Julia Rubin. 2021. Eqbench: A dataset of equivalent and non-equivalent program pairs. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 610–614. IEEE. find
  4. Ira D Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant’Anna, and Lorraine Bier. 1998. Clone detection using abstract syntax trees. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272), pages 368–377. IEEE. find
  5. Saikat Chakraborty, Shuvendu Lahiri, Sarah Fakhoury, Akash Lal, Madanlal Musuvathi, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, and Nikhil Swamy. 2023. Ranking llm-generated loop invariants for program verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pag… find
  6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 34 others. 2021. Evaluating l…
  7. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990–998. PMLR. find
  8. Yijiang Dong, Tiancheng Hu, and Nigel Collier. 2024. Can llm be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126–10141. find
  9. Aram Ebtekar. 2019. How to interpret contest ratings. Codeforces Blog. Accessed: 2025-05-18. find
  10. Mark Gabel, Lingxiao Jiang, and Zhendong Su. 2008. Scalable detection of semantic clones. In Proceedings of the 30th international conference on Software engineering, pages 321–330. find
  11. Robert Goldblatt and Marcel Jackson. 2012. Well-structured program equivalence is highly undecidable. ACM Transactions on Computational Logic (TOCL), 13(3):1–8. find
  12. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  13. Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246. — [arXiv:2503.02246](https://arxiv.org/abs/2503.02246) find
  14. Miguel Romero Rosas, Miguel Torres Sanchez, and Rudolf Eigenmann. 2024. Should ai optimize your code? a comparative study of current large language models versus classical optimizing compilers. arXiv preprint arXiv:2406.12146. — [arXiv:2406.12146](https://arxiv.org/abs/2406.12146) find
  15. Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2025. From llms to llm-based agents for software engineering: A survey of current, challenges and future. Preprint, arXiv:2408.02479. — [arXiv:2408.02479](https://arxiv.org/abs/2408.02479) find
  16. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in neural information processing systems, 33:20601– 20611.
  17. Heejung Kim, Yungbum Jung, Sunghun Kim, and Kwankeun Yi. 2011. Mecc: memory comparison-based clone detector. In Proceedings of the 33rd International Conference on Software Engineering, pages 301–310. find
  18. Kathryn T Stolee, Sebastian Elbaum, and Daniel Dobos. 2014. Solving the search for source code. ACM Transactions on Software Engineering and Methodology (TOSEM), 23(3):1–45. find
  19. Jens Krinke and Chaiyong Ragkhitwetsagul. 2022. Bigclonebench considered harmful for machine learning. In 2022 IEEE 16th International Workshop on Software Clones (IWSC), pages 1–7. IEEE. find
  20. Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated program repair. Communications of the ACM, 62(12):56–65. find
  21. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, and 11 others. 2022. Competition-lev…
  22. Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, and Hui Liu. 2025. Exploring the potential of general purpose llms in automated software refactoring: an empirical study. Automated Software Engineering, 32(1):26. find
  23. Nickil Maveli, Antonio Vergari, and Shay B Cohen. 2025. What can large language models capture about code functional equivalence? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6865–6903, Albuquerque, New Mexico. find
  24. Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. 2014. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 476– 480. IEEE. find
  25. Weixi Tong and Tianyi Zhang. 2024. Codejudge: Evaluating code generation with large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20032–20051. find
  26. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  27. Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yaofeng Sun, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, and 1 others. 2025. Equibench: Benchmarking code reasoning capabilities of large language models via equivalence checking. arXiv preprint arXiv:2502.12466. — [arXiv:2502.12466](https://arxiv.org/abs/2502.12466) find
  28. MikeMirzayanov. 2018. Codeforces: Problem difficulties. Codeforces Blog. Accessed: 2025-05-18. find
  29. Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 783– 794. find
  30. Kawser Wazed Nafi, Tonny Shekha Kar, Banani Roy, Chanchal Kumar Roy, and Kevin A. Schneider. 2019. Clcdsa: Cross language code clone detection using syntactical features and api documentation. 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1026– 1037. find
  31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems…
  32. Ruchi Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir A. Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of codin… find
  33. Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2232–2242. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oza-2026-llms-are-brittle-simple-code/