Elamur Research⌕
☰
home / papers / 2025 / SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

Paper · 2025 · Samuel Miserendino, Michele Wang, Tejal Patwardhan et al.

Summary

Most coding benchmarks, including SWE-Bench, test isolated problems that are easy to unit-test. They show little about how models handle realistic full-stack commercial work, or what that work is worth. The authors build SWE-Lancer from 1,488 real freelance jobs on the Expensify repository. These jobs were posted on Upwork and together paid about $1 million. The benchmark has two parts. In Individual Contributor (IC) SWE tasks, a model writes a patch, which is checked by browser-driven end-to-end tests that professional engineers wrote and reviewed. In SWE Manager tasks, the model chooses among competing freelancer proposals and is scored against the hiring manager's actual decision. Because every task has its real payment attached, results can be reported as dollars earned. The strongest model tested, Claude 3.5 Sonnet, still fails most tasks. The work matters because it ties model ability to concrete economic value. It also offers an evaluation that is harder to game than unit tests, and a public split is released for further research.

Key points

  • Of the tasks, 764 are IC SWE jobs carrying a combined value of $414,775, while the other 724 are SWE Manager jobs totaling $585,225. Individual payouts reach up to $32,000 for a single feature.
  • IC tasks are graded with Playwright end-to-end tests. About 100 professional engineers built these tests and validated each one three times. A separate engineer review agreed with 99% of the original managers' choices.
  • The public split, SWE-Lancer Diamond, holds $500,800 of tasks and comes with a unified Docker image. The remaining tasks are kept private to limit contamination.
  • An average Diamond task took 26 days to resolve and drew 47 comments. IC fixes touch about 2 files and 69 lines, and manager tasks offer 4–5 proposals to choose from.
  • Claude 3.5 Sonnet resolved 26.2% of IC tasks and 44.9% of manager tasks, earning $208,050 on Diamond and over $400,000 on the full set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksCode & programming languages · 194 works

Cite

Cite as Samuel Miserendino, Michele Wang, Tejal Patwardhan and Johannes Heidecke. 2025. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?. arXiv:2502.12115.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.12115 · arXiv 2502.12115

@misc{miserendino2025swe,
  title = {{SWE-Lancer: Can Frontier LLMs Earn \$1 Million from Real-World Freelance Software Engineering?}},
  author = {Samuel Miserendino, Michele Wang, Tejal Patwardhan and Johannes Heidecke},
  year = {2025},
  doi = {10.48550/arxiv.2502.12115},
  eprint = {2502.12115},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.12115}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Expensify. Issue \#41239: Add support for copy/pasting images on ios. https://github.com/Expensify/ App/issues/41239, 2024b. ↗
  2. Anthropic. Responsible scaling policy. Technical report, Anthropic, October 2024. URL https://assets.anthropic.com/ Gupta, V., Fernández-Crehuet, J. M., and Hanne, T. Freem/24a47b00f10301cd/original/ lancers in the software development process: A sys-Anthropic-Responsible-Scaling-Policy-2024-10-15.… — [doi:10.3390/pr8101215](https://doi.org/10.3390/pr8101215) ↗
  3. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108. 07732. ↗
  4. Brar, H. K. and Kaur, P. J. Differentiating integration testing and unit testing. In 2015 2nd International Conference on Computing for Sustainable Global Development (IN-DIACom), pp. 796–798, 2015. com/2227-9717/8/10/1215. find
  5. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring coding challenge competence with apps, 2021. URL https://arxiv.org/abs/ 2105.09938. ↗
  6. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974. ↗
  7. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… — Evaluating Large Language Models Trained on Code
  8. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770. ↗
  9. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., tau Yih, S. W., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation, 2022. URL https://arxiv.org/ abs/2211.11501. ↗
  10. Lam, M. S., Teoh, J., Landay, J. A., Heer, J., and Bernstein, M. S. Concept induction: Analyzing unstructured text with high-level concepts using lloom. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, pp. 1–28. ACM, 2024. doi: 10.1145/ 3613904.3642830. URL http:/… find
  11. Chowdhury, N., Aung, J., Shern, C. J., Jaffe, O., Sherburn, D., Starace, G., Mays, E., Dias, R., Aljubeh, M., Glaese, M., Jimenez, C. E., Yang, J., Liu, K., and Madry, A. Introducing swe-bench verified. arXiv preprint arXiv:2407.01489, 2024.
  12. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, DeepMind, G. Introducing the frontier C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., safety framework, 2024. URL https: Gowal…
  13. Expensify. Issue \#25889: Dev: Share code avatar differs from profile avatar. https://github.com/ Expensify/App/issues/25889, 2024a. ↗
  14. Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems, 2023. URL https://arxiv.org/abs/2306.03091. ↗
  15. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code un… find
  16. Microsoft. Playwright: Fast and reliable end-to-end testing for modern web apps. https://playwright.dev/, 2025. ↗
  17. Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models, 2024. URL https://arxiv.org/ abs/2308.07124. ↗
  18. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis, 2023. URL https://arxiv.org/abs/2203. 13474. ↗
  19. OpenAI. Openai preparedness framework, 2023. URL https://cdn.openai.com/ openai-preparedness-framework-beta. pdf. ↗
  20. Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., Ma, Y., Tong, H., Trinh, K., Tian, C., Wang, Z., Wu, B., Xiong, Y., Yin, S., Zhu, M., Lieret, K., Lu, Y., Liu, G., Du, Y., Tao, T., Press, O., Callan, J., Huerta, E., and Peng, H.… — SciCode: A Research Coding Benchmark Curated by Scientists
  21. Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J., Wu, X., Press, O., Muennighoff, N., Synnaeve, G., Narasimhan, K. R., Yang, D., Wang, S. I., and Press, O. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024. URL https://arxiv.org/abs/2410.03859. ↗
  22. Zhang, S., Zhao, H., Liu, X., Zheng, Q., Qi, Z., Gu, X., Zhang, X., Dong, Y., and Tang, J. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts, 2024. URL https://arxiv.org/abs/2405.04520. — [arXiv:2405.04520](https://arxiv.org/abs/2405.04520) ↗
  23. Zhong, R., Zhang, P., Li, S., Ahn, J., Klein, D., and Steinhardt, J. Goal driven discovery of distributional differences via language descriptions, 2023. URL https://arxiv.org/abs/2302.14233. ↗
  24. Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., Brunner, S., Gong, C., Hoang, T., Zebaze, A. R., Hong, X., Li, W.-D., Kaddour, J., Xu, M., Zhang, Z., Yadav, P., Jain, N., Gu, A., Cheng, Z., Liu, J., Liu, Q., Wang, Z., Lo, D., Hui, B., Mue… — BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
  25. OpenAI. Learning to reason with llms, 2024a. find
  26. OpenAI. Openai o3 and o3-mini—12 days of openai: Day 12, 2024b. find
  27. OpenAI. OpenAI Authentication API Documentation, 2025. URL https://platform.openai.com/ docs/api-reference/authentication. ↗
  28. Paul, R., Tsai, W. T., Chen, Y., Fan, C., Cao, Z., and Huang, H. End-to-end (e2e) testing and evaluation of high-assurance systems. In Pham, H. (ed.), Springer Handbook of Engineering Statistics, Springer Handbooks, pp. 439–467. Springer, London, London, 2006. ISBN 978-1-85233-806-0. doi: 10.1007/ 9… — [doi:10.1007/978-1-84628-288-1_24](https://doi.org/10.1007/978-1-84628-288-1_24) find
  29. Quan, S., Yang, J., Yu, B., Zheng, B., Liu, D., Yang, A., Ren, X., Gao, B., Miao, Y., Feng, Y., Wang, Z., Yang, J., Cui, Z., Fan, Y., Zhang, Y., Hui, B., Lin, J., and Qwen Team, A. G. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint ar…
  30. Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv.org/abs/2409.04109. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miserendino-2025-swe-lancer-frontier-llms-earn/