Elamur Research⌕
☰
home / papers / 2026 / CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions

CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions

Paper · 2026 · Jingwei Shi, Xinxiang Yin, Jing Huang et al.

Summary

Code benchmarks for LLMs often have test suites that miss subtle corner cases, so flawed programs are wrongly marked correct. CodeHacker addresses this with an LLM agent modeled on the "hack" practice of competitive programming. For a specific submission, it searches for an input that breaks it, using stress testing, anti-hash attacks and logic-targeted cases guided by code analysis. The agent only proposes inputs. Expected outputs come from a verified standard solution, which avoids hallucinated answers. Before attacking contestant code, a Calibration Phase has the agent probe and repair its own Validator and Checker. On CodeContests+ problems that need special judges, the added hack cases raise TNR sharply while TPR stays stable. The generated cases also work well as training data for RL, improving results on LiveCodeBench. The authors further release CodeHackerBench to measure how well models reason adversarially. The work matters because it makes judging stricter and gives a harder benchmark for testing models' reasoning.

Key points

  • A hack counts as successful only if three conditions hold: the input passes the Validator, the standard solution handles it correctly, and the target submission fails with WA, RE, TLE or MLE.
  • In the ablation study, the full agent reached a 51.40% hack success rate with GPT-5-mini. Removing the Refinement Loop caused the largest drop, to 46.60%. Without the Code Analyst it fell to 49.86%, and without Stress Test to 50.12%.
  • The Anti-Hash Generator was used for only about 0.4% of submissions (those relying on rolling hashes), but it broke all of them.
  • On 1000 Special Judge problems, TNR rose from 82.18% (CodeContests+ baseline) to 96.05%, and TPR rose from 95.34% to 96.38%.
  • Adding the refined validator alone slightly lowered TNR to 82.08%. This shows that invalid inputs had previously been rejecting some wrong solutions for the wrong reason. All evaluation runs offline in a local sandbox.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🏢MeituanBeijing · China17 works · 3,769 citations🎓Shanghai University of Finance and EconomicsShanghai · China13 works · 825 citations🎓Northwestern Polytechnical UniversityXi'an · China13 works · 67 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksSoftware · 92 works

Cite

Cite as Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao and Jinman Zhao. 2026. CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.108

@inproceedings{shi2026codehacker,
  title = {{CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions}},
  author = {Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao and Jinman Zhao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.108},
  url = {https://doi.org/10.18653/v1/2026.acl-long.108}
}

References

5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  2. Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, Yixuan Tan, Jiafu Tang, Yilin Tang, Junlei Wu, Qianyu Xiao, Can Zheng, Shouchen Zhou, Yuxiang Zhu, Yiming Huang, and 2 others. 2025. Can llms generate reliable test case gener… — [arXiv:2506.06821](https://arxiv.org/abs/2506.06821) find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  5. Codeforces. 2023. Codeforces command lines. https://codeforces.com/blog/entry/121114. \#\# Limitations ↗

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-codehacker-automated-test-case-generation/