Elamur Research⌕
☰
home / papers / 2025 / CodeContests+: High-Quality Test Case Generation for Competitive Programming

CodeContests+: High-Quality Test Case Generation for Competitive Programming

Paper · 2025 · Zihan Wang, Siyao Liu, Yang Sun et al.

Summary

Public competitive-programming problems usually come with statements and reference solutions, but platforms such as CodeForces keep their test cases private. Datasets for training and evaluating LLMs therefore rely on synthetic tests. Existing approaches, such as input mutation or asking an LLM to write inputs directly, have two weaknesses. They miss tricky and large-scale cases, and they often produce inputs that break the problem's own constraints.

The authors propose a Generator-Validator (G-V) agent system to address both issues:

  • A Generator agent writes a parameterized C++ input generator using testlib.
  • A Validator agent writes a program that checks each generated input against the stated constraints. Inputs that fail are sent back to the generator with explanations so it can fix them.

Applying this to the CodeContests problem set yields CodeContests+, which also adds custom checkers for problems with multiple valid answers. Measured against 1.72 million labelled submissions, the new tests judge solutions far more accurately than the originals. RL training on a filtered subset also improves downstream coding performance. The approach matters because it lets the large stock of public problems be turned into reliable RL data.

Key points

  • The authors estimate that about one-third of the original CodeContests test cases violate problem constraints, which causes both false positives and false negatives.
  • The Generator agent proposes roughly 20 commands per problem, covering small to large sizes and special corner-case types. The Validator's feedback loop repeats until all inputs pass.
  • At equal TPR/TNR thresholds, the 5x version of CodeContests+ yields almost twice as many reliable problems as CodeContests. The 1x version uses a quarter of the tests yet still gives over 80% more reliable problems.
  • CodeContests+-Verified keeps only problems with TPR and TNR ≥ 0.9. The dataset reports per-problem TPR and TNR so users can choose their own filtering thresholds. It is available on Hugging Face as ByteDance-Seed/Code-Contests-Plus.
  • In RL experiments, Qwen2.5-32B was trained with GRPO under DAPO settings and evaluated on LiveCodeBench (avg@15). Overall score rose from 0.622 with CodeContests to 0.637 with CodeContests+-Verified, with gains on all three difficulty levels (Medium: 0.786 → 0.812).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China335 works · 43,759 citations🏢ByteDanceBeijing · China54 works · 4,494 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksSoftware · 150 worksComputer Science Applications · 184 works

Cite

Cite as Zihan Wang, Siyao Liu, Yang Sun, Ming Ding and Hongyan Li. 2025. CodeContests+: High-Quality Test Case Generation for Competitive Programming. Findings of the Association for Computational Linguistics: EMNLP 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.299 · arXiv 2506.05817

@inproceedings{wang2025codecontests,
  title = {{CodeContests+: High-Quality Test Case Generation for Competitive Programming}},
  author = {Zihan Wang, Siyao Liu, Yang Sun, Ming Ding and Hongyan Li},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.299},
  eprint = {2506.05817},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.299}
}

References

19 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732.
  2. Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xiaochen Zuo, Chi Zhang, Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, and 253 others. 2025. Seed-thinking-v1.5: Advancing superb reasoning model…
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  4. Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, Bowen Li, Linyi Li, Boyi Liu, Jerry Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, and 35 others. 2024. FullStack Bench: Evaluating LLMs as full stack coders. Preprint, arXiv:… — [arXiv:2412.00535](https://arxiv.org/abs/2412.00535) find
  5. Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Mürk, and 5 others. 20… — Competitive Programming with Large Reasoning Models
  6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 180 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs…
  7. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benc…
  8. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live-CodeBench: Holistic and contamination free evaluation of large language models for code. In The Twelfth International Conference on Learning Representations… find
  9. Yichuan Jiang. 2016. A survey of task allocation and load balancing in distributed systems. IEEE Trans. Parallel Distributed Syst., 27(2):585–599. find
  10. Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. TACO: topics in algorithmic code generation dataset. CoRR, abs/2312.14852.
  11. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, and 7 o… — Competition-Level Code Generation with AlphaCode
  12. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by Chat-GPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Syste… find
  13. Mike Mirzayanov. 2005. Testlib. find
  14. Mike Mirzayanov, Oksana Pavlova, Pavel MAVRIN, Roman Melnikov, Andrew Plotnikov, Vladimir Par-fenov, and Andrew Stankevich. 2020. Codeforces as an educational platform for learning programming in digitalization. Olympiads in Informatics, 14(133-142):14. find
  15. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  16. Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. 2024. Can language models solve Olympiad programming? CoRR, abs/2404.10952.
  17. Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816.
  18. Qwen Team. 2024. Qwen2.5 technical report. find
  19. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. DAPO: An open-source LLM reinforcement learning s…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-codecontests-high-quality-test-case/