Elamur Research⌕
☰
home / papers / 2026 / Can We Predict Before Executing Machine Learning Agents?

Can We Predict Before Executing Machine Learning Agents?

Paper · 2026 · Jingsheng Zheng, Jintian Zhang, Yujie Luo et al.

Summary

ML agents usually improve their code through repeated generate-execute-feedback cycles. Every candidate has to be trained and run, so a single MLE-Bench run can take many hours. The authors ask whether an LLM can act as an implicit world model and judge which solution will perform better before any code runs. They define a task called Data-centric Solution Preference: the model receives two candidate solutions along with the task's description and a report on its data, then chooses the better candidate and gives a confidence level for that choice.

To test this, they collected real trajectories from the AIDE and AutoMind agents on MLE-bench. Experts filtered these into a pairwise preference corpus. Each input is enriched with a Verified Data Analysis Report: GPT-5.1 writes profiling code, the code runs in a sandbox, and a human checks the log before it is turned into prose.

Reasoning models predict winners clearly better than chance, and their confidence is well calibrated. Built into the FORE-AGENT system, this prediction step speeds up search and also improves results. Skipping execution in this way could make ML agents much cheaper to run.

Key points

  • Corpus: 18,438 pairwise comparisons covering 26 MLE-bench tasks in three domains.
    • CV: 5,952 pairs; NLP: 6,682 pairs; Data Science: 5,804 pairs.
    • The pairs come from 895 curated solutions, selected out of 1,329 valid ones produced by agents running on DeepSeek-V3.1 and o3-mini.
  • Prediction accuracy: DeepSeek-V3.2-Thinking reaches 61.5%, compared with 50.0% for random guessing and 50.8% for a complexity-based heuristic.
  • FORE-AGENT design: it uses a Predict-then-Verify loop. A confidence gate at c = 0.7 filters candidates, and only the top-1 candidate is actually executed.
  • FORE-AGENT results: tested against AIDE on 5 AI4Science tasks (2 of them unseen) under a 12-hour budget.
    • About 6× faster convergence.
    • 3.2× more search nodes explored.
    • +6% Beat Ratio.
    • +23% Test Improve Rate.
  • Release: code and dataset are available at github.com/zjunlp/predict-before-execute. The authors suggest the corpus could serve as a basis for training reward models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jingsheng Zheng, Jintian Zhang, Yujie Luo, Yuren Mao, Yunjun Gao, Lun Du, Huajun Chen and Ningyu Zhang. 2026. Can We Predict Before Executing Machine Learning Agents?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.182

@inproceedings{zheng2026can,
  title = {{Can We Predict Before Executing Machine Learning Agents?}},
  author = {Jingsheng Zheng, Jintian Zhang, Yujie Luo, Yuren Mao, Yunjun Gao, Lun Du, Huajun Chen and Ningyu Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.182},
  url = {https://doi.org/10.18653/v1/2026.acl-long.182}
}

References

97 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yash Akhauri, Xingyou Song, Arissa Wongpanich, Bryan Lewandowski, and Mohamed S. Abdelfattah. 2025. Regression language models for code. Preprint, arXiv:2509.26476. find
  2. Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar. 2025. Autoformulation of mathematical optimization models using llms. Preprint, arXiv:2411.01679. — [arXiv:2411.01679](https://arxiv.org/abs/2411.01679) find
  3. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
  4. Christian Cabrera, Andrei Paleyes, Pierre Thodoroff, and Neil D. Lawrence. 2025. Machine learning systems: A survey from a data-oriented perspective. Preprint, arXiv:2302.04810. — [arXiv:2302.04810](https://arxiv.org/abs/2302.04810) find
  5. Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xinyu Zhu, Mengcheng Zhou, Yanfeng Wang, Weinan E, Yuzhi Zhang, Linfeng Zhang, and Siheng Chen. 2025. Scimaster: Towards general-purpose scientific ai agents, part i. x-master as foundation: Can we lead on humanity’s last exam? Preprint, arXiv:2507.05241.
  6. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M adry. 2025. Mle-bench: Evaluating machine learning agents on machine learning engineering. Preprint, arXiv:2410.07095.
  7. Liu Changshu, Chen Yang, and Reyhaneh Jabbarvand. 2024. Codemind: Evaluating large language models for code reasoning. http://arxiv.org/abs/2402.09664. — [arXiv:2402.09664](https://arxiv.org/abs/2402.09664) ↗
  8. Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025a. Reasoning runtime behavior of a program with llm: How far are we? In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1869–1881. find
  9. Ke Chen, Peiran Wang, Yaoning Yu, Xianyang Zhan, and Haohan Wang. 2025b. Large language model-based data science agent: A survey. Preprint, arXiv:2508.02744. — [arXiv:2508.02744](https://arxiv.org/abs/2508.02744) find
  10. Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, Yimeng Zhang, Yihao Liang, Yuhang Zhou, Jiaqi Wang, Zhi Chen, and Wanxiang Che. 2025c. Ai4research: A survey of artificial intelligence for scientific research. Preprint, arXiv:…
  11. Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, Huaxiu Yao, Hengduo Li, Jiacheng Zhu, Xian Li, Dawn Song, Bo Li, Jason Weston, and Dat Huynh. 2025d. Scaling agent learning via experience synthesis. Preprint, arXiv:2511.03773. find
  12. Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, Bang Liu, and Chenglin Wu. 2024. Sela: Tree-search enhanced llm agents for automated machine learning. Preprint, arXiv:2410.17238. — [arXiv:2410.17238](https://arxiv.org/abs/2410.17238) find
  13. Alex O. Davies, Roussel Nzoyem, Nirav Ajmeri, and Telmo M. Silva Filho. 2025. Language models do not embed numbers continuously. Preprint, arXiv:2510.08009. — [arXiv:2510.08009](https://arxiv.org/abs/2510.08009) find
  14. DeepSeek-AI. 2024. Deepseek api documentation: Parameter settings. https://api-docs.deepseek.com/quick_start/parameter_settings . Accessed: 2025-12-21. ↗
  15. DeepSeek-AI. 2025a. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  16. DeepSeek-AI. 2025b. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  17. Jingtao Ding, Yunke Zhang, Yu Shang, Jie Feng, Yuheng Zhang, Zefang Zong, Yuan Yuan, Hongyuan Su, Nian Li, Jinghua Piao, Yucheng Deng, Nicholas Sukiennik, Chen Gao, Fengli Xu, and Yong Li. 2025. Understanding world or predicting future? a comprehensive survey of world models. Preprint, arXiv:2411.14… — [arXiv:2411.14499](https://arxiv.org/abs/2411.14499) find
  18. Anil R. Doshi and Oliver P. Hauser. 2024. Generative ai enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28):eadn5290. find
  19. Shangheng Du, Xiangchao Yan, Dengyang Jiang, Jiakang Yuan, Yusong Hu, Xin Li, Liang He, Bo Zhang, and Lei Bai. 2025. Automlgen: Navigating fine-grained optimization for coding agents. Preprint, arXiv:2510.08511. — [arXiv:2510.08511](https://arxiv.org/abs/2510.08511) find
  20. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. 2015. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636–638. find
  21. Haoyang Fang, Boran Han, Nick Erickson, Xiyuan Zhang, Su Zhou, Anirudh Dagar, Jiani Zhang, Ali Caner Turkmen, Cuixiong Hu, Huzefa Rangwala, Ying Nian Wu, Bernie Wang, and George Karypis. 2025. Mlzero: A multi-agent system for end-to-end machine learning automation. Preprint, arXiv:2505.13941. — [arXiv:2505.13941](https://arxiv.org/abs/2505.13941) find
  22. Jichen Feng, Yifan Zhang, Chenggong Zhang, Yifu Lu, Shilong Liu, and Mengdi Wang. 2025. Web world models. Preprint, arXiv:2512.23676. — [arXiv:2512.23676](https://arxiv.org/abs/2512.23676) find
  23. Luciano Floridi, Yiyang Jia, and Fernando Tohmé. 2025. A categorical analysis of large language models and why llms circumvent the symbol grounding problem. Preprint, arXiv:2512.09117. — [arXiv:2512.09117](https://arxiv.org/abs/2512.09117) find
  24. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, et al. 2025. Towards an ai co-scientist. Preprint, arXiv:2502.18864.
  25. Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri. 2024. Symbolic regression with a learned concept library. Preprint, arXiv:2409.09359. find
  26. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. 2024a. Cruxeval: A benchmark for code reasoning, understanding and execution. Preprint, arXiv:2401.03065.
  27. Yang Gu, Hengyu You, Jian Cao, Muran Yu, Haoran Fan, and Shiyou Qian. 2024b. Large language models for constructing and optimizing machine learning workflows: A survey. Preprint, arXiv:2411.10478. — [arXiv:2411.10478](https://arxiv.org/abs/2411.10478) find
  28. Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. Ds-agent: Automated data science by empowering large language models with case-based reasoning. Preprint, arXiv:2402.17453. find
  29. David Ha and Jürgen Schmidhuber. 2018. World models. Zenodo. find
  30. Annan Li, Chufan Wu, Zengle Ge, et al. 2025a. The fm agent. Preprint, arXiv:2510.26144. find
  31. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2024. Mastering diverse domains through world models. Preprint, arXiv:2301.04104. find
  32. Haoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Li Qing, and Lei Chen. 2025b. Exposing numeracy gaps: A benchmark to evaluate fundamental numerical abilities in large language models. In Findings of the Association for Computational Lingui… find
  33. Sirui Hong, Yizhang Lin, Bang Liu, et al. 2024. Data interpreter: An llm agent for data science. Preprint, arXiv:2402.18679.
  34. Andre Hora. 2024. Predicting test results without execution. https://doi.org/10.1145/3663529.3663794, pages 542–546. — [doi:10.1145/3663529.3663794](https://doi.org/10.1145/3663529.3663794) ↗
  35. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. Mlagentbench: Evaluating language agents on machine learning experimentation. Preprint, arXiv:2310.03302.
  36. Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. 2025. Deep research agents: A systematic examination and roadmap. Preprint, arXiv:2506.18096. — [arXiv:2506.18096](https://arxiv.org/abs/2506.18096) find
  37. Naman Jain, King Han, Alex Gu, Wending Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. Preprint, arXiv:2403.07974.
  38. Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. 2025c. Codei/o: Condensing reasoning patterns via code input-output prediction. Preprint, arXiv:2502.07316.
  39. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2025d. Mlr-copilot: Autonomous machine learning research based on large language models agents. Preprint, arXiv:2408.14033. find
  40. Yixia Li, Hongru Wang, Jiahao Qiu, Zhenfei Yin, Dongdong Zhang, Cheng Qian, Zeping Li, Pony Ma, Guanhua Chen, Heng Ji, and Mengdi Wang. 2025e. From word to world: Can large language models be implicit text-based world models? Preprint, arXiv:2512.18832. find
  41. Ruibo Liu, Jason Wei, Shixiang Shane Gu, Teyen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M. Dai. 2022. Mind’s eye: Grounded language model reasoning through simulation. Preprint, arXiv:2210.05359. find
  42. Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. 2025. Aide: Ai-driven exploration in the space of code. Preprint, arXiv:2502.13138. find
  43. Yixiu Liu, Yang Nan, Weixian Xu, Xiangkun Hu, Lyumanshan Ye, Zhen Qin, and Pengfei Liu. 2025a. Alphago moment for model architecture discovery. Preprint, arXiv:2507.18074. find
  44. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2024. Dsbench: How far are data science agents from becoming data science experts? arXiv preprint arXiv:2409.07703. find
  45. Zexi Liu, Yuzhu Cai, Xinyu Zhu, Yujie Zheng, Runkun Chen, Ying Wen, Yanfeng Wang, Weinan E, and Siheng Chen. 2025b. Ml-master: Towards ai-for-ai via integration of exploration and reasoning. Preprint, arXiv:2506.16499. — [arXiv:2506.16499](https://arxiv.org/abs/2506.16499) find
  46. Hoang Just, Ming Jin, Anit Sahu, Huy Phan, and Ruoxi Jia. 2024. Data-centric human preference optimization with rationales. arXiv (Cornell University). find
  47. Yujie Luo, Zhuoyun Yu, Xuehai Wang, Yuqi Zhu, Ningyu Zhang, Lanning Wei, Lun Du, Da Zheng, and Huajun Chen. 2025. Executable knowledge graphs for replicating ai research. Preprint, arXiv:2510.17795. — [arXiv:2510.17795](https://arxiv.org/abs/2510.17795) find
  48. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916.
  49. Stepan Kulibaba, Artem Dzhalilov, Roman Pakhomov, Oleg Svidchenko, Alexander Gasnikov, and Aleksei Shpilman. 2025. Kompeteai: Accelerated autonomous multi-agent system for end-to-end pipeline generation for machine learning problems. Preprint, arXiv:2508.10177. — [arXiv:2508.10177](https://arxiv.org/abs/2508.10177) find
  50. Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. 2025. Shinkaevolve: Towards open-ended and sample-efficient program evolution. Preprint, arXiv:2509.19349. find
  51. Rachel Metz. 2024. Openai scale ranks progress toward ’human-level’ ai. Bloomberg. find
  52. Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan Ö. Arık, and Tomas Pfister. 2025. Mle-star: Machine learning engineering agent via search and targeted refinement. Preprint, arXiv:2506.15692. — [arXiv:2506.15692](https://arxiv.org/abs/2506.15692) find
  53. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. 20… find
  54. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, et al. 2025. Alphaevolve: A coding agent for scientific and algorithmic discovery. Preprint, arXiv:2506.13131.
  55. OpenAI. 2024. Openai o1 system card. Preprint, arXiv:2412.16720.
  56. OpenAI. 2025a. System Card for gpt-5. Accessed on August 13, 2025. find
  57. OpenAI. 2025b. System Card for o3-mini. Accessed on December 11, 2025. find
  58. Yixin Ou, Yujie Luo, Jingsheng Zheng, Lanning Wei, Zhuoyun Yu, Shuofei Qiao, Jintian Zhang, Da Zheng, Yuren Mao, Yunjun Gao, Huajun Chen, and Ningyu Zhang. 2025. Automind: Adaptive knowledgeable agent for automated data science. Preprint, arXiv:2506.10974. — [arXiv:2506.10974](https://arxiv.org/abs/2506.10974) find
  59. Rushi Qiang, Yuchen Zhuang, Yinghao Li, Dingu Sagar V K, Rongzhi Zhang, Changhao Li, Ian Shuhei Wong, Sherry Yang, Percy Liang, Chao Zhang, and Bo Dai. 2025. Mle-dojo: Interactive environments for empowering llm agents in machine learning engineering. Preprint, arXiv:2505.07782. find
  60. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023. Reasoning with language model prompting: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5368–…
  61. Shuofei Qiao, Yanqiu Zhao, Zhisong Qiu, Xiaobin Wang, Jintian Zhang, Zhao Bin, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2025. Scaling generalist data-analytic agents. Preprint, arXiv:2509.25084. — [arXiv:2509.25084](https://arxiv.org/abs/2509.25084) find
  62. H. G. Rice. 1953. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2):358–366. find
  63. Christopher Michael Rytting and David Wingate. 2021. Leveraging the inductive bias of large language models for abstract textual reasoning. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA. Curran Associates Inc. find
  64. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using llm agents as research assistants. Preprint, arXiv:2501.04227.
  65. Judy Shen, Archit Sharma, Judy Shen, Qin Jun, Archit Sharma, and Jun Qin. 2024. Towards data-centric rlhf: Simple metrics for preference dataset comparison. http://arxiv.org/abs/2409.09603, abs/2409.09603. — [arXiv:2409.09603](https://arxiv.org/abs/2409.09603) ↗
  66. Lin Shi, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv (Cornell University). find
  67. Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. 2025a. Zerosearch: Incentivize the search capability of llms without searching. Preprint, arXiv:2505.04588.
  68. Ji Sun, Guoliang Li, Peiyao Zhou, Yihui Ma, Jingzhe Xu, and Yuan Li. 2025b. Agenticdata: An agentic data analytics system for heterogeneous data. Preprint, arXiv:2508.05002. — [arXiv:2508.05002](https://arxiv.org/abs/2508.05002) find
  69. FAIR CodeGen team, Jade Copet, Quentin Carbonneaux, et al. 2025. Cwm: An open-weights llm for research on code generation with world models. Preprint, arXiv:2510.02387. — [arXiv:2510.02387](https://arxiv.org/abs/2510.02387) find
  70. InternAgent Team, Bo Zhang, Shiyang Feng, et al. 2025. Internagent: When agent becomes the scientist – building closed-loop system from hypothesis to verification. Preprint, arXiv:2505.16938. find
  71. Edan Toledo, Karen Hambardzumyan, Martin Josifoski, et al. 2025. Ai research agents for machine learning: Search, exploration, and generalization in mle-bench. Preprint, arXiv:2507.02554. — [arXiv:2507.02554](https://arxiv.org/abs/2507.02554) find
  72. Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. 2025. Automl-agent: A multi-agent llm framework for full-pipeline automl. Preprint, arXiv:2410.02958. — [arXiv:2410.02958](https://arxiv.org/abs/2410.02958) find
  73. Andrej Tschalzev, Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. 2024. A data-centric perspective on evaluating machine learning models for tabular data. Preprint, arXiv:2407.02112. — [arXiv:2407.02112](https://arxiv.org/abs/2407.02112) find
  74. Sai Wang, Senthilnathan Subramanian, Mudit Sahni, Praneeth Gone, Lingjie Meng, Xiaochen Wang, Nicolas Ferradas Bertoli, Tingxian Cheng, and Jun Xu. 2025a. Configurable multi-agent framework for scalable and realistic testing of llm-based agents. Preprint, arXiv:2507.14705. — [arXiv:2507.14705](https://arxiv.org/abs/2507.14705) find
  75. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  76. Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago S. F. X. Teixeira, Diyi Yang, Ke Wang, and Alex Aiken. 2025a. Equibench: Benchmarking large language models’ reasoning about program semantics via equivalence checking. Preprint, arXiv:2502.12466.
  77. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  78. Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. 2025b. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. Preprint, arXiv:2502.18449. find
  79. Lionel Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash K. Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. 2023. From word models to world models: Translating from natural language to the probabilistic language of thought. Preprint, arXiv:2306.12672. find
  80. Xu Yang, Xiao Yang, Shikai Fang, Yifei Zhang, Jian Wang, Bowen Xian, Qizheng Li, Jingyuan Li, Minrui Xu, Yuante Li, Haoran Pan, Yuge Zhang, Weiqing Liu, Yelong Shen, Weizhu Chen, and Jiang Bian. 2025. R&d-agent: An llm-agent framework towards autonomous data science. Preprint, arXiv:2505.14738. — [arXiv:2505.14738](https://arxiv.org/abs/2505.14738) find
  81. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. Preprint, arXiv:2210.03629.
  82. Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, and Jiaxuan You. 2025a. Researchtown: Simulator of human research community. Preprint, arXiv:2412.17767.
  83. Zhaojian Yu, Kaiyue Feng, Yilun Zhao, Shilin He, Xiaoping Zhang, and Arman Cohan. 2025b. Alphare-search: Accelerating new algorithm discovery with language models. Preprint, arXiv:2511.08522. find
  84. Daochen Zha, Zaid Pervaiz Bhat, Kweiherng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. 2023. Data-centric artificial intelligence: A survey. Preprint, arXiv:2303.10158. — [arXiv:2303.10158](https://arxiv.org/abs/2303.10158) find
  85. Jiahuan Zhang, Tianheng Wang, Hanqing Wu, Ziyi Huang, Yulong Wu, Dongbai Chen, Linfeng Song, Yue Zhang, Guozheng Rao, and Kaicheng Yu. 2025a. Sr-llm: Rethinking the structured representation in large language model. Preprint, arXiv:2502.14352.
  86. Jintian Zhang, Kewei Xu, Jingsheng Zheng, Zhuoyun Yu, Yuqi Zhu, Yujie Luo, Lanning Wei, Shuofei Qiao, Lun Du, Da Zheng, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2025b. Innogym: Benchmarking the innovation potential of ai agents. Preprint, arXiv:2512.01822. — [arXiv:2512.01822](https://arxiv.org/abs/2512.01822) find
  87. Shaolei Zhang, Ju Fan, Meihao Fan, Guoliang Li, and Xiaoyong Du. 2025c. Deepanalyze: Agentic large language models for autonomous data science. Preprint, arXiv:2510.16872. — [arXiv:2510.16872](https://arxiv.org/abs/2510.16872) find
  88. Wenlin Zhang, Xiaopeng Li, Yingyi Zhang, Pengyue Jia, Yichao Wang, Huifeng Guo, Yong Liu, and Xiangyu Zhao. 2025d. Deep research: A survey of autonomous research agents. Preprint, arXiv:2508.12752. — [arXiv:2508.12752](https://arxiv.org/abs/2508.12752) find
  89. Xilin Zhang, Zhixin Mao, Ziwen Chen, and Shen Gao. 2024a. Effective tool augmented multi-agent framework for data analysis. DATA INTELLIGENCE, 6(4):923–945. find
  90. Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. 2024b. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. AAAI Conference on Artificial Intelligence. find
  91. Yuge Zhang, Qiyang Jiang, XingyuHan XingyuHan, Nan Chen, Yuqing Yang, and Kan Ren. 2024c. Benchmarking data science agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5677–5700. find
  92. Jiakang Yuan, Xiangchao Yan, Shiyang Feng, Bo Zhang, Tao Chen, Botian Shi, Wanli Ouyang, Yu Qiao, Lei Bai, and Bowen Zhou. 2025. Dolphin: Moving towards closed-loop auto-research through thinking, practice, and feedback. Preprint, arXiv:2501.03916. find
  93. Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2025e. Mlrc-bench: Can language agents solve machine learning research challenges? Preprint, arXiv:2504.09702. find
  94. Daojian Zeng, Lin Zhou, Zhiheng Zhang, and Lincheng Jiang. 2025. Autogen: Automated tool learning data generation with domain-specific structured data. DATA INTELLIGENCE, 7(4):1108–1128. find
  95. Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. 2025a. Ai scientists fail without strong implementation capability. Preprint, arXiv:2506.01372. find
  96. Liu Zexi, Jingyi Chai, Zexi Liu, Zhu Xinyu, Jingyi Chai, Tang Shuo, Xinyu Zhu, Ye Rui, Shuo Tang, Zhang Bo, Rui Ye, Bai Lei, Bo Zhang, Siheng Chen, Lei Bai, and Siheng Chen. 2025. Ml-agent: Reinforcing llm agents for autonomous machine learning engineering. https://doi.org/10.48550/arxiv.2505.23723,… — [doi:10.48550/arxiv.2505.23723](https://doi.org/10.48550/arxiv.2505.23723) ↗
  97. Yizhang Zhu, Liangwei Wang, Chenyu Yang, et al. 2025b. A survey of data agents: Emerging paradigm or overstated hype? Preprint, arXiv:2510.23587. — [arXiv:2510.23587](https://arxiv.org/abs/2510.23587) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-can-we-predict-before-executing/