Elamur Research⌕
☰
home / papers / 2023 / TACO: Topics in Algorithmic COde generation dataset

TACO: Topics in Algorithmic COde generation dataset

Paper · 2023 · Rongao Li, Jie Fu, Bo-Wen Zhang et al.

Summary

Benchmarks such as HumanEval and MBPP are close to saturation. They mostly test short, basic functions. Competition-style sets like APPS and CodeContests suffer from duplicates, weak test suites that let wrong code pass, and little detail on which algorithmic abilities a problem demands. TACO, from BAAI and collaborators, is a larger and harder resource for training and evaluating code generation. Problems were scraped from CodeChef, CodeForces, HackerRank and GeeksforGeeks and merged with APPS, CodeContests and Description2code. The authors then stripped comments, removed duplicate code, added extra test cases and checked solutions with unit tests. Each task carries metadata: difficulty, algorithm topics, skill types, time and memory limits, and expected complexity. In evaluation, even GPT-4 solves few of the hardest tasks, and open models do far worse. Fine-tuning on skill-specific subsets gives small gains on some categories. TACO therefore offers both a demanding benchmark and labels that support targeted diagnosis and training of code models.

Key points

  • Size: 26,443 problems (25,443 train, 1,000 test) with 1,539,152 verified Python 3 solutions, down from about 2.05M before deduplication.
    • Each problem has about 58 solutions on average.
    • Test cases average 51.6 per training problem and 202.3 per test problem.
  • Labels: 968 raw tags were merged into 36 algorithm topics, such as Complete Search and Dynamic Programming, plus separate skill-type labels.
  • Difficulty results: pass@1 falls sharply as problems get harder.
    • GPT-4 scores 31.5 on easy problems and 2.0 on very hard ones.
    • starcoder-15.5b scores 11.6 on easy and 0.18 on very hard.
  • Skill-specific fine-tuning: skill-specific LoRAs for starcoder-1b beat a single LoRA trained on the whole set for Data Structures, Greedy Algorithms and Sorting.
  • Availability and caveat: the data is released as BAAI/TACO on Hugging Face, with evaluation scripts in FlagOpen/TACO on GitHub. Because TACO overlaps with APPS and CodeContests, models trained on it should be evaluated on the TACO test set instead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China46 works · 8,682 citations🎓Shandong Normal UniversityJinan · China2 works · 157 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 168 works

Cite

Cite as Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin and Ge Li. 2023. TACO: Topics in Algorithmic COde generation dataset. arXiv:2312.14852.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.14852 · arXiv 2312.14852

@misc{li2023taco,
  title = {{TACO: Topics in Algorithmic COde generation dataset}},
  author = {Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin and Ge Li},
  year = {2023},
  doi = {10.48550/arxiv.2312.14852},
  eprint = {2312.14852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.14852}
}

References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  3. Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  5. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021.
  6. Antti Laaksonen. Competitive programmer’s handbook. Preprint, 5, 2017. find
  7. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  8. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. find
  9. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023.
  10. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. find
  11. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
  12. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D’efossez, Jade Copet, Fai… — Code Llama: Open Foundation Models for Code
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-taco-topics-algorithmic-code-generation/