Elamur Research⌕
☰
home / papers / 2025 / Measuring AI Ability to Complete Long Software Tasks

Measuring AI Ability to Complete Long Software Tasks

Paper · 2025 · Thomas Kwa, Ben West, Joel Becker et al.

Summary

Benchmark scores are hard to interpret in human terms, they saturate quickly, and they cannot be compared across models of very different strength. METR's authors address this with the 50% task-completion time horizon: how long expert humans need for tasks that an AI agent finishes successfully half the time. They built a 170-task suite from a subset of HCAST, RE-Bench, and a new set of very short tasks called SWAA. Skilled professionals were timed on most tasks. Frontier models released between 2019 and 2025 were run several times on every task. A logistic model, inspired by Item Response Theory, then links each model's success probability to the human completion time.

Frontier horizons have grown exponentially, doubling about every seven months, possibly faster since 2024. o3 reaches roughly 110 minutes. Gains seem to come from better reliability, error recovery, reasoning, and tool use. If the trend holds and transfers to real work, AI could handle month-long software tasks within about five years. This matters for forecasting and safety governance.

Key points

  • Task suite: 97 HCAST tasks (1 minute to 30 hours), 7 RE-Bench ML engineering tasks (8 hours each), and 66 SWAA single-step tasks (1–30 seconds). All are scored automatically.
  • Human baselines: more than 800 baselines totalling 2,529 hours, recorded by professionals averaging about 5 years of experience. A minority of HCAST tasks rely on researcher estimates instead.
  • Success vs. task length: model success drops strongly as human completion time rises (R² ≈ 0.83). The 80% horizon follows a similar trend but is about 5× shorter.
  • Forecast of month-long tasks: naive extrapolation puts a 1-month horizon (167 work hours) around mid-2029, with an 80% CI roughly mid-2028 to mid-2030. The 2024–2025 trend would bring this to around 2027.
  • Limitations: agents perform worse on "messier," less structured tasks. Validity checks with SWE-bench Verified and other experiments found no clear slowdown on more realistic tasks, but external validity remains uncertain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

Cite

Cite as Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499 · NeurIPS 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.14499 · arXiv 2503.14499

@misc{kwa2025measuring,
  title = {{Measuring AI Ability to Complete Long Software Tasks}},
  author = {Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney {Von Arx}, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan},
  year = {2025},
  doi = {10.48550/arxiv.2503.14499},
  eprint = {2503.14499},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.14499}
}

References

52 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  2. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024.
  3. Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.13793, 2024.
  4. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. find
  5. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025.
  6. Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. Artificial intelligence index report 2024, 2024. URL https://arxiv.org/abs/2… — [arXiv:2405.19522](https://arxiv.org/abs/2405.19522) ↗
  7. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified. https://openai.com/index/introducin… ↗
  8. David Rein, Joel Becker, Amy Deng, Seraphina Nix, Chris Canal, Daniel O’Connell, Pip Arnott, Ryan Bloom, Thomas Broadley, Katharyn Garcia, Brian Goodrich, Max Hasin, Sami Jawhar, Megan Kinniment, Thomas Kwa, Aron Lajko, Nate Rush, Lucas Jun Koba Sato, Sydney Von Arx, Ben West, Lawrence Chan, and Eli… find
  9. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as age… — AgentBench: Evaluating LLMs as Agents
  10. Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
  11. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= fibxvahvs3. ↗
  12. Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024.
  13. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024. ↗
  14. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
  15. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering, 2024. URL https://arxiv.org/abs/2410.07095. ↗
  16. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents to becoming data science experts? arXiv preprint arXiv:2409.07703, 2024.
  17. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?, 2025. URL https://arxiv.org/abs/2502.12115. ↗
  18. Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, and Simeon Campos. Mapping ai benchmark data to quantitative risk estimates through expert elicitation. arXiv preprint arXiv:2503.04299, 2025. — [arXiv:2503.04299](https://arxiv.org/abs/2503.04299) find
  19. David Owen. How predictable is language model benchmark performance?, 2024.
  20. Govind Pimpale, Axel Højmark, Jérémy Scheurer, and Marius Hobbhahn. Forecasting frontier language model agent capabilities, 2025. URL https://arxiv.org/abs/2502.15850. — [arXiv:2502.15850](https://arxiv.org/abs/2502.15850) ↗
  21. Richard Ngo. Clarifying and predicting AGI. https://www.lesswrong.com/posts/ BoA3agdkAzL6HQtQP/clarifying-and-predicting-agi, 2023. Accessed: 2024-03-21. ↗
  22. Joseph Carlsmith. How much computational power does it take to match the human brain?, 2020. URL https://www.openphilanthropy.org/research/ how-much-computational-power-does-it-take-to-match-the-human-brain/. Open Philanthropy report, August 14, 2020. ↗
  23. Ajeya Cotra. Draft report on ai timelines, 2020. URL https://www.alignmentforum.org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines. Draft report on AI timelines; technical report. ↗
  24. Frank B Baker. The basics of item response theory. ERIC, 2001. find
  25. R. J. de Ayala. Handbook of Item Response Theory: Models, Applications, and Issues. Guilford Press, 2017. find
  26. Fernando Martı́nez-Plumed, Ricardo B.C. Prudêncio, Adolfo Martı́nez-Usó, and José Hernández-Orallo. Item response theory in ai: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271:18–42, 2019. ISSN 0004-3702. doi: https://doi.org/10. 1016/j.artint.2018.09.0… find
  27. OpenAI. Openai o3-mini. https://openai.com/index/openai-o3-mini, 2025. [Accessed 18-03-2025]. ↗
  28. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. ↗
  29. Dave Zielinski. How to optimize onboarding, June 2019. URL https://www.shrm.org/topics-tools/news/hr-magazine/how-to-optimize-onboarding. Accessed on March 17, 2025. ↗
  30. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2018. URL https://gluebenchmark.com/. ↗
  31. Alex Wang, Yada Pruksachatkun, Naman Nangia, Amanpreet Singh, Julian Michael, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2019. URL https://super.gluebenchmark.com/. ↗
  32. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR, abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300. ↗
  33. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world… — ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
  34. Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Shiyang Chen, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin Shin, Qiaochu Yang, Anh… — [arXiv:2502.09696](https://arxiv.org/abs/2502.09696) find
  35. [Author] Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  36. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. arXiv preprint, arXiv:2107.03374. ↗
  37. Jeremy Austin, Augustus Odena, et al. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. arXiv preprint, arXiv:2108.07732. ↗
  38. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. URL htt…
  39. Dario Amodei and Danny Hernandez. Ai and compute, 2018. URL https://openai.com/ blog/ai-and-compute/. OpenAI blog post, May 16, 2018. ↗
  40. Epoch AI. Data on notable AI models, 2024. notable-ai-models. Accessed: 2025-03-04. URL https://epoch.ai/data/ ↗
  41. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022. find
  42. Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926, 2024. find
  43. Hao Song and Peter Flach. Efficient and robust model benchmarks with item response theory and adaptive testing. International Journal of Interactive Multimedia and Artificial Intelligence, 2021. find
  44. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAM-BADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for C… find
  45. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  46. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https://arxiv.org/abs/2210.03629. ↗
  47. Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench – a sparse benchmark of reasoning and knowledge in large language models, 2025. URL https://arxiv.org/abs/2407.12844. ↗
  48. Ege Erdil and Tamay Besiroglu. Algorithmic progress in computer vision, 2023. URL https://arxiv.org/abs/2212.05153. — [arXiv:2212.05153](https://arxiv.org/abs/2212.05153) ↗
  49. Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models, 2024. URL https://arxiv.org/abs/2403.05812. — [arXiv:2403.05812](https://arxiv.org/abs/2403.05812) ↗
  50. Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138, 2025.
  51. METR. Measuring automated kernel engineering. https://metr.org/blog/ 2025-02-14-measuring-automated-kernel-engineering/, 02 2025. ↗
  52. David Malin Roodman. On the probability distribution of long-term changes in the growth rate of the global economy: An outside view, 2020. URL https://api.semanticscholar.org/CorpusID:221641799. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kwa-2025-measuring-ai-ability-complete/