Elamur Research⌕
☰
home / papers / 2025 / Competitive Programming with Large Reasoning Models

Competitive Programming with Large Reasoning Models

Paper · 2025 · OpenAI: Ahmed El-Kishky, Alexander Wei, Andre Saraiva et al.

Summary

This OpenAI report asks whether LLM performance in competitive programming improves more from hand-built, task-specific inference pipelines or from scaling general reinforcement learning. It compares three systems. The first is o1, a reasoning model trained with RL to produce long chains of thought and to run code in a sandbox. The second is o1-ioi, which continues o1's RL on hard C++ problems and adds an AlphaCode-like test-time pipeline: mass sampling, clustering, learned reranking and controlled submission. The third is an early o3 checkpoint that uses no engineered strategy. The models are evaluated on simulated CodeForces Division 1 contests held after the training cut-off and screened for contamination with embeddings. They are also tested at the live IOI 2024. There, o1-ioi reached only the 49th percentile and earned gold only under relaxed limits. By contrast, o3 obtained gold under standard rules and a CodeForces rating comparable to elite humans. The takeaway is that general RL scaling lets effective reasoning strategies emerge by themselves, which beats human-designed heuristics.

Key points

  • CodeForces ratings rise sharply with RL-based reasoning: gpt-4o scored 808 (11th percentile), o1-preview 1258 (62nd) and o1 1673 (89th).
  • o1-ioi scored 1807 (93rd percentile) on its own. Filtering out solutions that failed public tests raised this to 2092 (96th), and the full test-time strategy reached 2214 (98th).
  • The o1-ioi pipeline works as follows:
    • It splits each IOI problem into subtasks and draws 10,000 samples per subtask.
    • It clusters programs by their outputs on 256 model-generated inputs, keeping inputs that pass at least 75% of model-written validators.
    • It reranks with a learned scorer plus penalties whose weights were tuned on past IOI problems.
  • At IOI 2024, o1-ioi competed under human conditions (six problems, ten hours, up to 50 submissions per problem) and scored 213 points, placing in the 49th percentile.
  • o3 reached IOI 2024 gold without hand-crafted domain strategies. The authors also report that scaling both RL compute and test-time compute steadily improves results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Science Applications · 131 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as OpenAI: Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, Oleg Murk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju and Wenda Zhou. 2025. Competitive Programming with Large Reasoning Models. arXiv:2502.06807.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.06807 · arXiv 2502.06807

@misc{elkishky2025competitive,
  title = {{Competitive Programming with Large Reasoning Models}},
  author = {OpenAI: Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, Oleg Murk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju and Wenda Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2502.06807},
  eprint = {2502.06807},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.06807}
}

References

16 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  3. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  4. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  5. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
  6. Leblond, Rémi and Gimeno, Felix and Altché, Florent and Saade, Alaa and Ruddock, Anton and Tallec, Corentin and Powell, George and Grill, Jean-Bastien and Miku la, Maciej and Lochbrunner, Matthias and others. Alphacode 2 technical report. https://storage.googleapis.com/deepmind-media/AlphaCode2/Al… ↗
  7. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  8. Mike Mirzayanov. Codeforces rating system. https://codeforces.com/blog/entry/102, 2010. ↗
  9. Mike Mirzayanov. Open codeforces rating system. https://codeforces.com/blog/entry/20762, 2016. ↗
  10. Mike Mirzayanov. Codeforces: Soon we will change the rating calculation for new accounts. https://codeforces.com/blog/entry/77890, 2020. ↗
  11. OpenAI. Introducing swe-bench verified. https://openai.com/index/introducing-swe-bench-verified/, August 2024. Accessed: 2025-01-14. ↗
  12. OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, September 2024. Accessed: 2025-01-14. ↗
  13. OpenAI. Openai o3 system card. Technical Report, 2025. find
  14. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023.
  15. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.
  16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/el-kishky-2025-competitive-programming-large-reasoning-models/