Elamur Research⌕
☰
home / papers / 2026 / CodeRise: Bootstrapping LLMs for Ultra Low-Resource Programming Languages via Progressive Self-Refinement Curriculum

CodeRise: Bootstrapping LLMs for Ultra Low-Resource Programming Languages via Progressive Self-Refinement Curriculum

Paper · 2026 · Tengfei Wen, Xuanang Chen, Ben He et al.

Summary

Code LLMs work well for Python and increasingly for languages like R or Perl, but they perform poorly on ultra low-resource programming languages (ULRPLs), where public code is scarce and hard to collect. Self-Instruct-style synthesis (e.g., SelfCodeAlign) breaks down in this setting for two reasons. Models often cannot produce any valid program (the cold-start problem), and the samples they do produce skew toward generic tasks such as string handling. CodeRise addresses this with a two-stage pipeline.

  1. Stage 1 puts the language's full formal syntax in context and samples standard-library modules with inverse-frequency weighting. This yields a sandbox-verified warm-up set for a first fine-tune.
  2. Stage 2 removes the long syntax prompt. It then raises task difficulty through an adaptive curriculum and repairs failed solutions via multi-turn self-debugging with execution feedback.

The authors test CodeRise on Tengo and Janet. Models fine-tuned on its data beat both training-free and training-based baselines. The work matters because it shows a way to bootstrap code ability when no strong teacher model exists. The authors also release new evaluation suites for both languages.

Key points

  • Benchmarks: for Tengo, the authors migrated HumanEval-Tengo and MBPP-Tengo and built the new TengoEval; for Janet, they built the new JanetEval. Code and data are available at github.com/tengfei2000/CodeRise.
  • Long-tail problem: in 30,000 Tengo samples generated with SelfCodeAlign, 12,971 called library APIs. Among those, the text module accounted for about 47.8% of calls, while modules such as os and hex were almost absent.
  • Module sampling weights: modules are sampled with weight 1/log(cᵢ+2)^p, where cᵢ is how often module i appears in correct solutions. The Stage 2 difficulty controller mixes easy, medium and hard tasks at a 0.25/0.50/0.25 ratio.
  • Ablations:
    • Stage 2 self-refinement is the main source of gains.
    • Of the Stage 2 components, the curriculum matters most.
    • Removing self-debugging costs 4.2 points on HumanEval.
    • Module balancing and documentation retrieval mainly help library coverage on TengoEval rather than HumanEval. Stage 1 + Retrieval alone reaches 61.0 on HumanEval and 50.0 on TengoEval.
  • Warm-up length: within roughly 200 fine-tuning steps (batch size 8), the Tengo execution error rate falls from nearly 100% to below 30%. Training for about 200–300 steps achieves this while largely preserving general (HumanEval-Python) ability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🏢ByteDanceBeijing · China47 works · 4,339 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations🔬Institute of SoftwareBeijing · China27 works · 4,980 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksCode & programming languages · 168 worksLow-resource languages · 287 works

Cite

Cite as Tengfei Wen, Xuanang Chen, Ben He, Xiaoliang Cong and Le Sun. 2026. CodeRise: Bootstrapping LLMs for Ultra Low-Resource Programming Languages via Progressive Self-Refinement Curriculum. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1840

@inproceedings{wen2026coderise,
  title = {{CodeRise: Bootstrapping LLMs for Ultra Low-Resource Programming Languages via Progressive Self-Refinement Curriculum}},
  author = {Tengfei Wen, Xuanang Chen, Ben He, Xiaoliang Cong and Le Sun},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1840},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1840}
}

References

48 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732.
  2. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. Preprint, arXiv:2207.14255.
  3. Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q. Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. 2024. Knowledge transfer from high-resource to low-resource programming languages for code llms. Proc. ACM Program. Lang… find
  4. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2022. Multipl-e: A scalable and extensible approach to benchmarking neural code gener…
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, and 1 others. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.
  6. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  7. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  8. Yifeng Ding, Hantian Ding, Shiqi Wang, Qing Sun, Varun Kumar, and Zijian Wang. 2024. Horizon-length prediction: Advancing fill-in-the-middle capabilities for code generation with lookahead planning. CoRR, abs/2410.03103. — [arXiv:2410.03103](https://arxiv.org/abs/2410.03103) find
  9. Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun. 2023. Baldur: Whole-proof generation and repair with large language models. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco,…
  10. Alessandro Giagnorio, Alberto Martin-Lopez, and Gabriele Bavota. 2025. Enhancing code generation for low-resource languages: No silver bullet. Preprint, arXiv:2501.19085. — [arXiv:2501.19085](https://arxiv.org/abs/2501.19085) find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  12. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, and 1 others. 2024. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. Preprint, arXiv:2401.14196. find
  13. Hojae Han, Jaejin Kim, Jaeseok Yoo, Youngwon Lee, and Seung-won Hwang. 2024. Archcode: Incorporating software requirements in code generation with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bang… find
  14. Jamie Hayes, Ilia Shumailov, William P. Porter, and Aneesh Pappu. 2025. Measuring memorization in RLHF for code completion. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  15. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems.
  16. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, and Others. 2024. Qwen2.5-coder technical report. CoRR, abs/2409.12186.
  17. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May… find
  18. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? CoRR, abs/2310.06770.
  19. Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard. 2024. A survey on llm-based code generation for low-resource and domain-specific programming languages. Preprint, arXiv:2410.03981. — [arXiv:2410.03981](https://arxiv.org/abs/2410.03981) find
  20. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2023. The stack: 3 TB of permissively licensed source code. find
  21. Trans. Mach. Learn. Res., 2023. find
  22. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolu… find
  23. Kaixin Li, Qisheng Hu, James Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Michael Shieh, and Junxian He. 2024. Instructcoder: Instruction tuning large language models for code editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Student Researc… find
  24. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and… find
  25. Bingchang Liu, Chaoyu Chen, Zi Gong, Cong Liao, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, Wei Jiang, Hang Yu, and Jianguo Li. 2024. Mftcoder: Boosting code llms with multitask fine-tuning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data… find
  26. Federico Mora, Justin Wong, Haley Lepe, Sahil Bhatia, Karim Elmaaroufi, George Varghese, Joseph E. Gonzalez, Elizabeth Polgreen, and Sanjit A. Seshia. 2024. Synthetic programming elicitation for text-to-code in very low-resource programming and formal languages. In The Thirty-eighth Annual Conferenc… find
  27. Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2024. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, ICLR 202… find
  28. OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, and 1 others. 2024a. Openai o1 system card. Preprint, arXiv:2412.16720.
  29. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, and 1 others. 2024b. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  30. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  31. Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the 46… find
  32. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. find
  33. Foundations and Trends® in Information Retrieval, 3(4):333–389. Training open-source llms for effective and efficient github issue resolution. CoRR, abs/2501.05040. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, and 1 others. 2024. Code llama: Open found… — Code Llama: Open Foundation Models for Code
  34. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austri…
  35. Marc Szafraniec, Baptiste Rozière, Hugh Leather, Patrick Labatut, François Charton, and Gabriel Synnaeve. 2023. Code translation with compiler representations. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  36. Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, and Maosong Sun. 2024. DebugBench: Evaluating debugging capability of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4173– 4198,… find
  37. Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin, and 1 others. 2025. Epicoder: Encompassing diversity and complexity in code generation. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net. find
  38. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  39. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  40. Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Zachary Mueller, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. 2024a. Selfcodealign: Self-alignment for code generation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Inform…
  41. Min Xue, Artur Andrzejak, and Marla Leuther. 2024. An interpretable error correction method for enhancing code-to-code translation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Review.net. find
  42. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Preprint, arXiv:2405.15793.
  43. Xinran Yu, Chun Li, Minxue Pan, and Xuandong Li. 2024a. Droidcoder: Enhanced android code completion with context-enriched retrieval-augmented generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - Novem… find
  44. Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2024b. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti… find
  45. Lin Zheng, Jianbo Yuan, Zhi Zhang, Hongxia Yang, and Lingpeng Kong. 2024. Self-infilling code generation. Preprint, arXiv:2311.17972. — [arXiv:2311.17972](https://arxiv.org/abs/2311.17972) find
  46. Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Debug like a human: A large language model debugger via verifying runtime execution step by step. In Findings of the Association for Computational Linguistics: ACL 2024, pages 851–870, Bangkok, Thailand. Association for Computational Linguistics.
  47. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024b. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  48. Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. 2025. Swe-fixer: find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wen-2026-coderise-bootstrapping-llms-ultra-low/