Elamur Research⌕
☰
home / papers / 2026 / Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

Paper · 2026 · Fei Wang, Li Shen, Liang Ding et al.

Summary

Zeroth-Order (ZO) fine-tuning of LLMs needs only forward passes and therefore uses about as much memory as inference. In practice it converges slowly in wall-clock time and its gradient estimates are noisy. The authors profiled ZO training on OPT-6.7B and found that two steps, generating perturbations and applying parameter updates, take more than 40% of each training step. They also describe what they call "Policy Blindness": standard ZO methods perturb every layer equally, even though layers differ in how sensitive they are.

Their method, AdaLeZO, treats layer choice as a non-stationary Multi-Armed Bandit. Each step perturbs and updates only a sampled subset of layers. Layers are drawn with replacement, and a count-aware Inverse Probability Weighting estimator keeps the gradient estimate unbiased while lowering its variance.

Tests on LLaMA and OPT models from 6.7B to 30B parameters show 1.7–3.0× wall-clock speedups over strong baselines, with accuracy matched or improved and no extra memory. The conclusion cites gains of up to 5.1×. AdaLeZO also works as a plug-in for other ZO optimizers, which makes memory-limited fine-tuning more practical.

Key points

  • Bottleneck: Perturbation generation and updates scale linearly with the number of parameters and account for roughly 40–50% of per-step time for MeZO-style training on OPT-6.7B.
  • Plug-in results: AdaLeZO was added to four existing ZO optimizers, including LoZO and HiZOO. AdaHiZOO runs 2.0× faster than HiZOO and raises average accuracy by 0.15% on OPT-6.7B. Scaling results cover OPT-13B and OPT-30B, where AdaLeZO again beats MeZO.
  • Ablations (LLaMA-2-7b, SST-2 and BoolQ):
    • Sampling ratio ρ ≈ 0.2 outperforms full-parameter perturbation (ρ = 1.0).
    • An IPW clipping threshold of C = 4 works best. With C ≥ 12, BoolQ accuracy falls from 74.37% to 69.97%.
    • Softmax temperature τ = 0.6 works best.
    • An exploration coefficient γ = 0.1 adds 2.4% on BoolQ.
    • Adaptive selection beats uniform random sparse sampling by 2.12% on average.
  • Limitations: A precision gap to backpropagation remains on hard reasoning tasks such as DROP. Evaluation covers only supervised NLP fine-tuning; multimodal models and RLHF are untested.
  • Code: Available at github.com/WangFei-2019/UnifiedZO.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of SydneySydney · Australia45 works · 68,350 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations🎓South China University of TechnologyGuangzhou · China22 works · 1,554 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

Cite

Cite as Fei Wang, Li Shen, Liang Ding, Chao Xue, Ye Liu and Changxing Ding. 2026. Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1999

@inproceedings{wang2026universally,
  title = {{Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling}},
  author = {Fei Wang, Li Shen, Liang Ding, Chao Xue, Ye Liu and Changxing Ding},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1999},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1999}
}

References

62 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Although AdaLeZO delivers substantial improvements in wall-clock efficiency and convergence stability, several inherent limitations of the Zeroth-Order optimization paradigm remain. find
  2. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language pro…
  3. Performance Gap with First-Order Methods. AdaLeZO consistently surpasses standard zeroth-order baselines such as MeZO and narrows the gap with full fine-tuning. However, its performance does not fully reach that of first-order methods Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong De… — Qwen Technical Report
  4. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1. find
  5. Djallel Bouneffouf and Raphael Feraud. 2025. Multi-armed bandits meet large language models. arXiv preprint arXiv:2505.13355. — [arXiv:2505.13355](https://arxiv.org/abs/2505.13355) find
  6. Aichen Cai, Anmeng Zhang, Anyu Li, Bo Zhang, Bohua Cai, Chang Li, Changjian Jiang, Changkai Lu, Chao Xue, Chaocai Liang, Cheng Zhang, Dongkai Liu, Fei Wang, Guoqiang Huang, Haijian Ke, Han Lin, Hao Wang, Ji Miao, Jiacheng Zhang, and 50 others. 2026. Joyai-llm flash: Advancing mid-scale llms with tok… — [arXiv:2604.03044](https://arxiv.org/abs/2604.03044) find
  7. Taha Ceritli, Savas Ozkan, Jeongwon Min, Eunchung Noh, Cho Jung Min, and Mete Ozay. 2024. A study of parameter efficient fine-tuning by learning to efficiently fine-tune. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15819–15836. find
  8. Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. 2024. Deepzero: Scaling up zeroth-order optimization for deep model training. In The Twelfth International Conference on Learning Representat… find
  9. Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. 2019. Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization. Advances in neural information processing systems, 32. find
  10. Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. 2025. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In The Thirteenth International Conference on Learning Representations. find
  11. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. Association for Computational Linguistics. find
  12. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124.
  13. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024. Deepseek llm: Scaling open-source language models with longterm… find
  14. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.
  15. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic…
  16. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR. find
  17. Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. 2024. Variance-reduced zeroth-order methods for fine-tuning language models. In Proceedings of the 41st International Conference on Machine Learning, pages 15180–15208. find
  18. Saeed Ghadimi and Guanghui Lan. 2013. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368. find
  19. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  20. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  21. Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. 2024. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p… find
  22. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
  23. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations, pages 1– 13.
  24. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. find
  25. Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  26. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  27. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. LISA: Layerwise importance sampling for memory-efficient large language model fine-tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  28. Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, and Zhiyu Li. 2024. Controllable text generation for large language models: A survey. arXiv preprint arXiv:2408.12599.
  29. Yiqi Lin and Ru Wang. 2023. Bandit-nas: Bandit sampling method for neural architecture search. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  30. Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. 2018. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications. In International Conference on Artificial Intelligence and Statistics, pages 288–297. PMLR. find
  31. Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. 2020. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43–54. find
  32. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61–6…
  33. Yuxi Liu, Renjia Deng, Yutong He, Xue Wang, Tao Yao, and Kun Yuan. 2025. MISA: Memory-efficient LLMs optimization with module-wise importance sampling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  34. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075. find
  35. Yurii Nesterov and Vladimir Spokoiny. 2017. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566. find
  36. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating c…
  37. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
  38. Jun Rao, Zepeng Lin, Xuebo Liu, Xiaopeng Ke, Lian Lian, Dong Jin, Shengjun Cheng, Jun Yu, and Min Zhang. 2025. APT: Improving specialist LLM performance with weakness case acquisition and iterative preference training. In Findings of the Association for Computational Linguistics: ACL 2025, pages 209… find
  39. Jun Rao, Xuebo Liu, Hexuan Deng, Zepeng Lin, Zixiong Yu, Jiansheng Wei, Xiaojun Meng, and Min Zhang. 2026. Dynamic sampling that adapts: Self-aware iterative data persistent optimization for mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2026.
  40. Jun Rao, Xuebo Liu, Lian Lian, Shengjun Cheng, Yunjie Liao, and Min Zhang. 2024. CommonIT: Commonality-aware instruction tuning for large language models via data partitions. In EMNLP, pages 10064–10083, Miami, Florida, USA. Association for Computational Linguistics. find
  41. Jun Rao, Fei Wang, Liang Ding, Shuhan Qi, Yibing Zhan, Weifeng Liu, and Dacheng Tao. 2022. Where does the performance improvement come from - a reproducibility concern about image-text retrieval. In SIGIR. find
  42. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
  43. Hyunseok Seung, Jaewoo Lee, and Hyunsuk Ko. 2025. Low-rank curvature for zeroth-order optimization in llm fine-tuning. arXiv preprint arXiv:2511.07971. — [arXiv:2511.07971](https://arxiv.org/abs/2511.07971) find
  44. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631…
  45. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885.
  46. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations. find
  47. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trai…
  48. Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, and Dacheng Tao. 2025. Tezo: Empowering the low-rankness on the temporal dimension in the zeroth-order optimization for fine-tuning llms. arXiv preprint arXiv:2501.19057. find
  49. Qitao Tan, Jun Liu, Zheng Zhan, Caiwen Ding, Yanzhi Wang, Xiaolong Ma, Jaewoo Lee, Jin Lu, and Geng Yuan. 2025. Harmony in divergence: Towards fast, accurate, and memory-efficient zeroth-order LLM fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  51. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  52. Fei Wang, Li Shen, Liang Ding, Chao Xue, Ye Liu, and Changxing Ding. 2024. Simultaneous computation and memory efficient zeroth-order optimizer for fine-tuning large language models. arXiv preprint arXiv:2410.09823. — [arXiv:2410.09823](https://arxiv.org/abs/2410.09823) find
  53. Fei Wang, Li Shen, Liang Ding, Chao Xue, Ye Liu, and Changxing Ding. 2025. Layer as puzzle pieces: Compressing large language models through layer concatenation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  54. Ziming Yu, Pan Zhou, Sike Wang, Jia Li, Mi Tian, and Hua Huang. 2025. Zeroth-order fine-tuning of llms in random subspaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4475–4485. find
  55. Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. 2024. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Forty-first International… find
  56. Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, and Zheng Zhang. 2025. Mazo: Masked zeroth-order optimization for multi-task fine-tuning of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pa… find
  57. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning. find
  58. Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor Tsang. 2025. Second-order fine-tuning without pain for LLMs: A hessian informed zeroth-order optimizer. In The Thirteenth International Conference on Learning Representations. find
  59. Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, and Zheng Zhang. 2025. QuZO: Quantized zeroth-order fine-tuning for large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages… find
  60. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765–2781.
  61. Pengyun Yue, Xuanlin Yang, Mingqing Xiao, and Zhouchen Lin. 2025. PseuZO: Pseudo-zeroth-order algorithm for training deep neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. *Table 3: Hyperparameter Settings. We list the common settings shared across all… find
  62. LOZO Rank r Interval v HiZOO Estimate times n Smooth scale α DiZO Projection Update Cycle Projection Iterations Smoothing Scalar Projection Step Size Clip Range PseuZO Sliding Window L Cycles / Epochs Descent Formula 14 2 / 10 λ max λ(t) = 1+10t Sampling Ratio ρ Temperature τ Exploration Coeff. γ EM… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-universally-empowering-zeroth-order-optimization/