Elamur Research⌕
☰
home / papers / 2025 / Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs

Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs

Paper · 2025 · Liang Zeng, Yongcong Li, Yuzhen Xiao et al.

Summary

Training LLM agents to fix real GitHub issues is held back by scarce data. Every task needs an executable environment and validated unit tests, and building these by hand is slow, so most public SWE datasets have only a few thousand instances. The authors created an automated, incremental pipeline with three stages: collecting and pre-filtering candidate pull requests, validating them by execution in Docker runtime images, and generating agent trajectories. The result is the Skywork-SWE dataset of 10,169 Python tasks from 2,531 repositories. Each task has a natural-language description and its own environment image for automatic test checking. They fine-tuned Qwen2.5-Coder-32B inside the OpenHands framework on more than 8,000 successful trajectories. Performance on SWE-bench Verified rose steadily, roughly log-linearly, as training data grew, with no plateau in sight. This is evidence that data scaling laws apply to software engineering agents and that execution-grounded data is the main bottleneck. The released Skywork-SWE-32B model narrows the gap between open and proprietary systems.

Key points

  • Dataset scale: Skywork-SWE contains 10,169 runtime-validated Python instances from 2,531 repositories. Repositories that appear in SWE-bench Verified were excluded to avoid leakage.
  • Single-attempt result: After fine-tuning on 8,209 trajectories, Skywork-SWE-32B reaches 38.0% pass@1 on SWE-bench Verified with a single attempt, no verifier and OpenHands v0.32.0. This is a new best among Qwen2.5-Coder-32B-based OpenHands agents.
  • Test-time scaling: With test-time scaling the score rises to 47.0%. This beats earlier sub-32B results, such as Devstral-Small-2505 at 46.8%.
  • Practical lessons:
    • Environment images for the 500 Verified tasks need about 1,000 GB of disk.
    • Running rollout and validation in mini-batches lets the same images be reused for both steps.
    • Results vary noticeably between OpenHands versions.
    • Trajectories with more than 50 interaction rounds can exceed the model's 32K-token context.
  • Release and future work: The model weights are published on Hugging Face (Skywork/Skywork-SWE-32B). Proposed next steps are multilingual benchmarks such as Multi-SWE-Bench and online RL that uses test-verified rewards.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksCode & programming languages · 129 works

Cite

Cite as Liang Zeng, Yongcong Li, Yuzhen Xiao, Changshi Li, Chris Yuhao Liu, Rui Yan, Tianwen Wei, Jujie He, Xuchen Song, Yang Liu and Yahui Zhou. 2025. Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs. arXiv:2506.19290.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.19290 · arXiv 2506.19290

@misc{zeng2025skywork,
  title = {{Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs}},
  author = {Liang Zeng, Yongcong Li, Yuzhen Xiao, Changshi Li, Chris Yuhao Liu, Rui Yan, Tianwen Wei, Jujie He, Xuchen Song, Yang Liu and Yahui Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2506.19290},
  eprint = {2506.19290},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.19290}
}

References

60 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Claude 3.5 sonnet announcement. https://www.anthropic.com/news/claude-3-5-sonnet , 2024a. Accessed: 2025-06-04. ↗
  2. Anthropic. Claude haiku 3.5. https://www.anthropic.com/claude/haiku , 2024b. Accessed: 2025-06-03. ↗
  3. Anthropic. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/claude-3-7-sonnet , 2025. Accessed: 2025-06-04. ↗
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  5. Ibragim Badertdinov, Maria Trofimova, Yuri Anapolskiy, Sergey Abramov, Karina Zainullina, Alexander Golubev, Sergey Polezhaev, Daria Litvintseva, Simon Karasik, Filipp Fisin, Sergey Skvortsov, Maxim Nekrashevich, Anton Shevtsov, and Boris Yangel. Scaling data collection for training software enginee… find
  6. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024.
  7. ByteDance Seed. Doubao-1.5-pro. https://seed.bytedance.com/zh/doubao_1_5_pro , 2025. Accessed: 2025-06-04. ↗
  8. Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-v0: Train real-world long-horizon agents via reinforcement learni… find
  9. Chi In Chang, Wan Chong Choi, and Iek Chong Choi. A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course. In Proceedings of the 2024 7th International Conference on Big Data and Education, pages 29–35, 2024. find
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  11. DeepSeek AI. Deepseek coder: Let the code write itself. https://deepseekcoder.github.io/ , 2024. Accessed: 2025-05-13. ↗
  12. DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412.19437 . ↗
  13. Google DeepMind. Gemini 2.0 flash. https://deepmind.google/technologies/gemini , 2024. Accessed: 2025-06-04. ↗
  14. Google DeepMind. Gemini 2.5 pro. https://deepmind.google/models/gemini/pro/ , 2025. Accessed: 2025-06-04. ↗
  15. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  16. Shousheng Jia Haosheng Zou, Xiaowei Lv and Xiangzheng Zhang. 360-llama-factory, 2024. URL https://github.com/Qihoo360/360-LLaMA-Factory . ↗
  17. Jordan Hoffmann, Sebastian Biette, Jacopo de Aiken, James Lei, Zi Lin Chen, Jeff Clune, Li Zhang, Jose Suarez, Frank M. Hutter, Greg Brockman, Jack W. Rae, Lei Chinchilla, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022a. URL https://arxiv.org/abs/2203.15… ↗
  18. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022b.
  19. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024.
  20. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  21. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
  22. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515, 2024.
  23. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66 . ↗
  24. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020a.
  25. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Scott Gray, Alec Radford, Dario Amodei, and Ilya Sutskever. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020b. URL https://arxiv.org/abs/2001.08361 . ↗
  26. Rémi Leblond, Felix Gimeno, Florent Altché, Alaa Saade, Anton Ruddock, Corentin Tallec, George Powell, Jean-Bastien Grill, Maciej Mikuła, Matthias Lochbrunner, et al. Alphacode 2 technical report. Technical report, DeepMind, December 2023. Accessed: 2025-01-14. find
  27. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. find
  28. Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021.
  29. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. find
  30. Yingwei Ma, Yongbin Li, Yihong Dong, Xue Jiang, Rongyu Cao, Jue Chen, Fei Huang, and Binhua Li. Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute. arXiv preprint arXiv:2503.23803, 2025. — [arXiv:2503.23803](https://arxiv.org/abs/2503.23803) find
  31. Mistral AI. Devstral: Introducing the best open-source model for coding agents. https://mistral.ai/news/devstral , 2025. Accessed: 2025-06-03. ↗
  32. OpenAI. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/ , 2024. Accessed: 2025-06-04. ↗
  33. OpenAI. Hello gpt-4o, 2024a. URL https://openai.com/index/hello-gpt-4o/ . Accessed: 2025-05-30. ↗
  34. OpenAI. Introducing openai o1, 2024b. URL https://openai.com/o1/ . Accessed: 2025-05-30. ↗
  35. OpenAI. Introducing openai o1 preview, introducing-openai-o1-preview/ . Accessed: 2025-05-30. find
  36. OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/ , September 2024d. Accessed: 2025-01-14. ↗
  37. OpenAI. Swe-bench verified: A human-validated subset for ai model evaluation. https://openai.com/index/introducing-swe-bench-verified , 2024e. Accessed: 2025-05-13. ↗
  38. OpenAI. Openai o3-mini. https://openai.com/index/openai-o3-mini/ , 2025. Accessed: 2025-06-04. ↗
  39. OpenAI. Openai o3 and o4-mini https://cdn.openai.com/pdf/April 2025. 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf , Accessed: 2025-05-13. ↗
  40. Arvid Orwall. Moatless tools: Practical large-codebase editing with language models. https://github.com/aorwall/moatless-tools , 2024. Commit \<hash\> accessed May 2025. ↗
  41. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024. find
  42. Python Software Foundation. The python package index (pypi), 2023. URL https://pypi.org/ . ↗
  43. Qwen. Qwen2.5-coder series: Powerful, diverse, practical. https://qwenlm.github.io/blog/qwen2.5-max/ , 2024a. Accessed: 2025-06-04. ↗
  44. Qwen. Qwen2.5-max: Maximizing reasoning and tool use abilities. https://qwenlm.github.io/blog/qwen2.5-max/ , 2024b. Accessed: 2025-06-04. ↗
  45. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. 2024c.
  46. URL — HybridFlow: A Flexible and Efficient RLHF Framework https://openai.com/index/system card. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. ↗
  47. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  48. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.
  49. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/ . torchtune maintainers and contributors. torchtune: Pytorch’s finetuning library, April 2024. URL https//github.com/pytorch/torchtune . ↗
  50. Haoran Wang, Zhenyu Hou, Yao Wei, Jie Tang, and Yuxiao Dong. Swe-dev: Building software engineering agents with training and inference scaling. https://ubecwang.notion.site/1bc32cf963e080b2a01df2895f66021f?v=1bc32cf963e0810ca07e000c86c4c1e1 , 2025. Accessed: 2025-05-13. ↗
  51. Xingyao Wang. ing and critic Sota model, on swe-bench verified with inference-time scal-2025. URL https://www.all-hands.dev/blog/sota-on-swe-bench-verified-with-inference-time-scaling-and-critic-model . ↗
  52. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2024.
  53. Lilian Weng. Llm-powered autonomous agents. lilianweng.github.io, Jun 2023. URL https://lilianweng.github.io/posts/2023-06-23-agent/ . ↗
  54. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024. find
  55. Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source llms for effective and efficient github issue resolution. arXiv preprint arXiv:2501.05040, 2025. find
  56. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024.
  57. John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798, 2025. find
  58. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: A multilingual benchmark for issue resolving, 2025… — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) find
  59. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1592–1604, New York, NY, USA, 2024. Association for Computing Machinery. IS… — [doi:10.1145/3650212.3680384](https://doi.org/10.1145/3650212.3680384) find
  60. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2025-skywork-swe-unveiling-data/