Elamur Research
☰
home / papers / 2021 / Measuring Coding Challenge Competence With APPS

Measuring Coding Challenge Competence With APPS

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2105.09938

Summary

APPS (Automated Programming Progress Standard) is a benchmark that tests whether language models can turn a free-form English problem statement into working Python code. This is close to how human programmers are tested in job interviews and programming contests. The authors collected 10,000 problems from open coding-practice sites such as Codewars, AtCoder, Kattis and Codeforces. They cleaned the problems with a separate parser for each site, removed near-duplicates and sorted them into three difficulty levels: introductory, interview and competition. Each generated program is graded by running it on test cases, not by comparing its text to a reference with metrics like BLEU. The authors fine-tuned GPT-2 and GPT-Neo on GitHub code and on the APPS training split, and ran GPT-3 in a few-shot setting. Overall scores are low and drop as problems get harder. Scores rise with model size and fine-tuning, and syntax errors fall sharply as models get stronger. BLEU proved misleading and sometimes ranked weaker models higher. APPS gives an execution-based, reproducible way to track progress in program synthesis.

Key points

  • The dataset has 10,000 problems, split evenly into 5,000 for training and 5,000 for testing. It includes 131,777 test cases and 232,421 human-written solutions. Problem statements average 293.2 words, and test problems have about 21.2 test cases each. The data is available at github.com/hendrycks/apps.
  • GPT-Neo 2.7B was the best model the authors tested on introductory problems. It passed an average of 14.7% of test cases with one attempt (Top-1) and 19.9% with the best of five attempts (Top-5). It fully solved only 3.9% (Top-1) and 5.5% (Top-5) of problems (strict accuracy).
  • GPT-3 175B in a few-shot setting solved only 3 of the 5,000 test problems, so it did worse than much smaller fine-tuned models. About 59% of its introductory solutions had syntax errors, compared with about 3% for GPT-Neo.
  • BLEU went up as problems got harder, even though real accuracy went down. BLEU also scored GPT-2 0.1B about the same as or higher than GPT-Neo 2.7B, which shows it is a poor stand-in for test-case evaluation.
  • In later work, OpenAI Codex (12B) scored 4.14% Top-1 and 9.65% Top-5 on introductory problems. With 1,000 samples per problem it reached about 25%, which is better but still far from solving the benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Topics: Machine translation

Cited by

296 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 267 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.09938 · arXiv 2105.09938

@misc{hendrycks2021measuring,
  title = {{Measuring Coding Challenge Competence With APPS}},
  author = {Hendrycks, Dan and Basart, Steven and Kadavath, Saurav and Mazeika, Mantas and Arora, Akul and Guo, Ethan and Burns, Collin and Puranik, Samir and He, Horace and Song, Dawn and Steinhardt, Jacob},
  year = {2021},
  doi = {10.48550/arxiv.2105.09938},
  eprint = {2105.09938},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.09938}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 207–216. find
  2. IEEE, 2013. Rajeev Alur, Dana Fisman, Saswat Padhi, Rishabh Singh, and Abhishek Udupa. find
  3. Sygus-comp 2018: Results and analysis. find
  4. SYNT, 2018. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Ma… find
  5. W. Ling, P. Blunsom, Edward Grefenstette, K. Hermann, Tomás Kociský, Fumin Wang, and A. Senior. Latent predictor networks for code generation. ArXiv, abs/1603.06744, 2016. find
  6. Hui Liu, Mingzhu Shen, Jiaqi Zhu, Nan Niu, Ge Li, and Lu Zhang. Deep learning based program generation from requirements text: Are we there yet? IEEE Transactions on Software Engineering, 2020a. find
  7. J. Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2020b. find our card
  8. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019. find
  9. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, A. Blanco, C. Clément, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, L. Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, N. Duan, N. Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning b… find
  10. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation. In International Conference on Automated Software Engineering (ASE), 2015. find
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. find our card
  12. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. ArXiv, abs/2009.03393, 2020. find our card
  13. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  14. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. find
  15. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020. find
  16. Veselin Raychev, Pavol Bielik, and Martin T. Vechev. Probabilistic model for code with decision trees. Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, 2016. find
  17. Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. Compositional program synthesis from natural language and examples. In IJCAI, 2015. find
  18. Shuo Ren, Daya Guo, Shuai Lu, L. Zhou, Shujie Liu, Duyu Tang, M. Zhou, A. Blanco, and S. Ma. Codebleu: a method for automatic evaluation of code synthesis. ArXiv, abs/2009.10297, 2020. — [arXiv:2009.10297](https://arxiv.org/abs/2009.10297) find
  19. L. Tang and R. Mooney. Using multiple clause constructors in inductive logic programming for semantic parsing. In ECML, 2001. find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017. find
  21. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019a. find our card
  22. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019b. find our card
  23. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942, 2019c. find
  24. Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. ArXiv, abs/2005.10636, 2020. find
  25. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887, 2018. find our card
  26. Maksym Zavershynskyi, A. Skidanov, and Illia Polosukhin. Naps: Natural program synthesis dataset. 2nd Workshop on Neural Abstract Machines and Program Induction, 2018. J. Zelle and R. Mooney. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, Vol. 2, 1996. Rowan Zell… find
  27. Programming Language Test Cases Number of Programs Lines per Program (Avg.) Number of Exercises Text Input Hearthstone Python 5 665 7.7 665 Card Text Django Python 5 18,805 1 18,805 Comment NAPS UAST APPS Python 17,477 21.7 2,231 Pseudocode 232,421 18.0 10,000 Problem Descriptions *Table 4: Further… find