Elamur Research⌕
☰
home / papers / 2026 / What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search

What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search

Paper · 2026 · Xinhao Zhang, Xi Chen, François Portet et al.

Summary

LLMs are often used as mutation operators inside evolutionary or agentic optimization loops, but it is unclear why some models drive search much better than others. The authors ran a controlled study of LLM-guided evolutionary search. They collected trajectories from 15 LLMs on 8 tasks in four domains: route optimization, prompt optimization, equation discovery and algorithmic problems. This produced roughly 72K candidate solutions.

Zero-shot ability correlates with final fitness but leaves much of the variance unexplained. Models that start at the same level can end up far apart. To find out why, the authors embedded candidates in semantic space and tracked entropy and dispersion across generations. Strong optimizers act as local refiners: they gain fitness in many small steps while their search narrows around promising regions. Weak optimizers wander semantically, make occasional jumps and then stall. Novelty on its own does not predict success. It helps only when the search stays localized.

This matters for practice. Choosing or training models for refinement behaviour, rather than raw capability, can yield cheaper and better optimizers.

Key points

  • Experimental loop. The setup follows Novikov et al. (2025). Every model starts from the same fixed initial population. Parents are drawn from the top 20% (q = 0.2) with probability proportional to fitness, and invalid outputs receive zero fitness.
  • Tasks:
    • TSP with 30 and 60 cities, scored by inverse tour length.
    • Prompt optimization for a frozen gpt-4o-mini on SAMSum (ROUGE-L) and ASSET (SARI).
    • Symbolic regression on Oscillator 1 and Oscillator 2 from Shojaee et al. (2025a).
  • Same start, different outcomes. On TSP-60, Gemini-1.5-Pro and Mistral-7B-Instruct have similar zero-shot scores. Gemini-1.5-Pro keeps improving and localizes its search, while Mistral-7B-Instruct stays novel without gaining fitness.
  • Supporting analyses:
    • A generation-level mixed-effects analysis shows that novelty raises the chance of a breakthrough only under low semantic dispersion.
    • Perturbation experiments that mix models to weaken refinement lead to predictable drops in performance.
  • Cost and released data. On the gain-versus-cost plot, Mistral-24B-Instruct sits on the Pareto frontier, showing that cheaper models can be efficient operators. The trajectories are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputational Theory and Mathematics · 252 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xinhao Zhang, Xi Chen, François Portet and Maxime Peyrard. 2026. What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1252

@inproceedings{zhang2026what,
  title = {{What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search}},
  author = {Xinhao Zhang, Xi Chen, François Portet and Maxime Peyrard},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1252},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1252}
}

References

55 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikhil Abhyankar, Sanchit Kabra, Saaketh Desai, and Chandan K. Reddy. 2026. Llema: Evolutionary search with llms for multi-objective materials discovery. Preprint, arXiv:2510.22503. — [arXiv:2510.22503](https://arxiv.org/abs/2510.22503) find
  2. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. Gepa: Reflective prompt evoluti… find
  3. Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. 2020. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computa…
  4. Huan ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, and 8 others. 2026. A survey of self-evolving agents: What, w… find
  5. Shuvayan Brahmachary, Subodh M. Joshi, Aniruddha Panda, Kaushik Koneripalli, Arun Kumar Sagotra, Harshil Patel, Ankush Sharma, Ameya D. Jagtap, and Kaushic Kalyanaraman. 2024. Large language model-based evolutionary optimizer: Reasoning with elitism. Preprint, arXiv:2403.02054. — [arXiv:2403.02054](https://arxiv.org/abs/2403.02054) find
  6. Xiangsen Chen, Ruilong Wu, Yanyan Lan, Ting Ma, and Yang Liu. 2026. Molevolve: Llm-guided evolutionary search for interpretable molecular optimization. Preprint, arXiv:2603.24382. — [arXiv:2603.24382](https://arxiv.org/abs/2603.24382) find
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… find
  8. Alex Duchnowski, Ellie Pavlick, and Alexander Koller. 2025. A knapsack by any other name: Presentation impacts LLM performance on NP-hard problems. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6628–6651, Suzhou, China. Association for Computational Linguistics. find
  9. Jordan S. Ellenberg, Cristofero S. Fraser-Taliente, Thomas R. Harvey, Karan Srivastava, and Andrew V. Sutherland. 2025. Generative modeling for mathematical discovery. Preprint, arXiv:2503.11061. — [arXiv:2503.11061](https://arxiv.org/abs/2503.11061) find
  10. Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. 2024. NPHardEval: Dynamic benchmark on reasoning ability of large language models via complexity classes. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 40…
  11. Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, Zhaochun Ren, Nikos Aletras, Xi Wang, Han Zhou, and Zaiqiao Meng. 2025. A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic sys… — [arXiv:2508.07407](https://arxiv.org/abs/2508.07407) find
  12. Mingxuan Yuan Xi Lin Fu Luo Zhenkun Wang Zhichao Lu Qingfu Zhang Fei Liu, Xialiang Tong. 2024. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In International Conference on Machine Learning (ICML). find
  13. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. Preprint, arXiv:2309.16797.
  14. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  15. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, and 15 others. 202… — Accelerating scientific discovery with Co-Scientist
  16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  17. Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri. 2024. Symbolic regression with a learned concept library. In Advances in Neural Information Processing Systems, volume 37, pages 44678–44709. Curran Associates, Inc. find
  18. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2025. Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers. Preprint, arXiv:2309.08532. find
  19. John H. Holland. 1992. Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. find
  20. Beichen Huang, Xingyu Wu, Yu Zhou, Jibin Wu, Liang Feng, Ran Cheng, and Kay Chen Tan. 2024. Exploring the true potential: Evaluating the black-box optimization capability of large language models. Preprint, arXiv:2404.06290. — [arXiv:2404.06290](https://arxiv.org/abs/2404.06290) find
  21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  22. Martin Josifoski, Lars Klein, Maxime Peyrard, Nicolas Baldwin, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and 1 others. 2023. Flows: Building blocks of reasoning and collaborating ai. arXiv preprint arXiv:2308.01285. — [arXiv:2308.01285](https://arxiv.org/abs/2308.01285) find
  23. Hyomin Lee, Sangwoo Park, Yumin Choi, Sohyun An, Seanie Lee, and Sung Ju Hwang. 2026. T-map: Red-teaming llm agents with trajectory-aware evolutionary search. Preprint, arXiv:2603.22341. — [arXiv:2603.22341](https://arxiv.org/abs/2603.22341) find
  24. Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. 2022. Evolution through large models. Preprint, arXiv:2206.08896. — [arXiv:2206.08896](https://arxiv.org/abs/2206.08896) find
  25. Xiaozhe Li, Jixuan Chen, Xinyu Fang, Shengyuan Ding, Haodong Duan, Qingwen Liu, and Kai Chen. 2025. Opt-bench: Evaluating llm agent on large-scale search spaces optimization problems. Preprint, arXiv:2506.10764. — [arXiv:2506.10764](https://arxiv.org/abs/2506.10764) find
  26. Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Hongzhang Liu, Ronghao Chen, Yangfan He, Daxin Jiang, Binxing Jiao, Chen Hu, and Huacan Wang. 2025. Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents. Preprint, arXiv:2508.020… — [arXiv:2508.02085](https://arxiv.org/abs/2508.02085) find
  27. Fei Liu, Qingfu Zhang, Jialong Shi, Xialiang Tong, Kun Mao, and Mingxuan Yuan. 2025. Fitness landscape of large language model-assisted automated algorithm search. Preprint, arXiv:2504.19636. — [arXiv:2504.19636](https://arxiv.org/abs/2504.19636) find
  28. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. Preprint, arXiv:2408.06292.
  29. Robert MacKnight, Jose Emilio Regio, Jeffrey G. Ethier, Luke A. Baldwin, and Gabe Gomes. 2025. Pre-trained knowledge elevates large language models beyond traditional chemical reaction optimizers. Preprint, arXiv:2509.00103. — [arXiv:2509.00103](https://arxiv.org/abs/2509.00103) find
  30. Mistral-AI, :, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Sagar Vaze, and 82 others.… find
  31. Melanie Mitchell and Charles E Taylor. 1999. Evolutionary computation: an overview. Annual Review of Ecology and Systematics, pages 593–616. find
  32. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Ba… — AlphaEvolve: A coding agent for scientific and algorithmic discovery
  33. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  34. Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. 2025. Kernelbench: Can llms write efficient gpu kernels? Preprint, arXiv:2502.10517.
  35. Maxime Peyrard, Martin Josifoski, and Robert West. 2025. Agentic ai: The era of semantic decoding. Preprint, arXiv:2403.14562. — [arXiv:2403.14562](https://arxiv.org/abs/2403.14562) find
  36. Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, Jiacheng Zhu, Xuan Jiang, Sirui Li, Cathy Wu, Bryan Kian Hsiang Low, Jinhua Zhao, and Paul Pu Liang. 2026. Coral: Towards autonomous multi-agent evolution for open-ended disco… find
  37. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2023. Mathematical discoveries from program search with large language models. N…
  38. Cosmin Paduraru, Christina Sorokin, and 1118 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.
  39. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  40. Niki van Stein, Haoran Yin, Anna V. Kononova, Thomas Bäck, and Gabriela Ochoa. 2025. Behaviour space analysis of llm-driven meta-heuristic discovery. Preprint, arXiv:2507.03605. — [arXiv:2507.03605](https://arxiv.org/abs/2507.03605) find
  41. Xingyu Wu, Sheng-Hao Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. 2025. Evolutionary computation in the era of large language model: Survey and roadmap. IEEE Transactions on Evolutionary Computation, 29(2):534–554. find
  42. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. Preprint, arXiv:2309.03409.
  43. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. 2025a. Llm-sr: Scientific equation discovery via programming with large language models. Preprint, arXiv:2404.18400. find
  44. Xianliang Yang, Ling Zhang, Haolong Qian, Lei Song, and Jiang Bian. 2025a. Heuragenix: Leveraging llms for solving complex combinatorial optimization challenges. Preprint, arXiv:2506.15196. — [arXiv:2506.15196](https://arxiv.org/abs/2506.15196) find
  45. Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. 2025b. Llm-srbench: A new benchmark for scientific equation discovery with large language models. Preprint, arXiv:2504.10415. find
  46. Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. 2025b. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. Preprint, arXiv:2410.07076. find
  47. Jianyong Sun, Hu Zhang, Qingfu Zhang, and Huanhuan Chen. 2018. Balancing exploration and exploitation in multiobjective evolutionary optimization. In Proceedings of the genetic and evolutionary computation conference companion, pages 199–200. find
  48. Anja Šurina, Amin Mansouri, Lars C.P.M. Quaedvlieg, Amal Seddas, Maryna Viazovska, Emmanuel Abbe, and Caglar Gulcehre. 2025. Algorithm discovery with LLMs: Evolutionary search meets reinforcement learning. In Second Conference on Language Modeling. find
  49. Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. 2024. Reevo: Large language models as hyper-heuristics with reflective evolution. In Advances in Neural Information Processing Systems, volume 37, pages 43571–43608. Curran Associates, Inc. find
  50. Shaohua Yu, Tianyu Chen, and Linyan Liu. 2026. Large language model-driven full-component evolution of adaptive large neighborhood search. Preprint, arXiv:2603.06996. — [arXiv:2603.06996](https://arxiv.org/abs/2603.06996) find
  51. Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. 2024. A survey on self-evolution of large language models. Preprint, arXiv:2404.14387. — [arXiv:2404.14387](https://arxiv.org/abs/2404.14387) find
  52. Leyi Zhao, Weijie Huang, Yitong Guo, Jiang Bian, Chenghong Wang, and Xuhong Zhang. 2026. Large language model-powered evolutionary code optimization on a phylogenetic tree. Preprint, arXiv:2601.14523. find
  53. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Zhikai Zhao, Chuanbo Hua, Federico Berto, Kanghoon Lee, Ziha… — [arXiv:2505.04480](https://arxiv.org/abs/2505.04480) find
  54. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024. Hypothesis generation with large language models. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), page 117–139. Association for Computational Linguistics. find
  55. Yongxin Zhou, Changshun Wu, Philippe Mulhem, Didier Schwab, and Maxime Peyrard. 2026. What matters to an LLM? behavioral and computational evidences from summarization. In Findings of the Association for Computational Linguistics: EACL 2026, pages 5712–5737, Rabat, Morocco. Association for Computati…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-what-makes-llm-good-optimizer/