Elamur Research⌕
☰
home / papers / 2026 / RoZO: Geometry-Aware Zeroth-Order Fine-Tuning on Low-Rank Adapters for Black-Box Large Language Models

RoZO: Geometry-Aware Zeroth-Order Fine-Tuning on Low-Rank Adapters for Black-Box Large Language Models

Paper · 2026 · Zichen Song and Weijia Li

Summary

Fine-tuning large language models with back-propagation needs a lot of memory, because activations must be stored. It is also impossible when only model outputs are accessible. Zeroth-order (ZO) methods avoid this by estimating gradients from loss differences under random perturbations, but these estimates are noisy. The earlier low-rank method LOZO draws perturbations from arbitrary random subspaces and ignores the structure of LoRA adapters. It also becomes unstable late in training. The authors propose RoZO, which treats LoRA updates as points on a manifold of fixed-rank matrices and perturbs only within its tangent space. Parallel transport carries momentum between iterations, and an SVD-based retraction keeps iterates at the target rank. A trust region adapts the perturbation radius, and a low-rank preconditioner provides Adam-style steps at almost no memory cost. The paper proves convergence with a tighter variance bound than LOZO's. On RoBERTa-large and OPT models up to 66B, RoZO outperforms MeZO and LOZO at equal query budgets and rivals first-order tuning when training data is scarce. This makes memory-light or black-box adaptation more practical.

Key points

  • RoZO and its momentum variant RoZO-M are compared against MeZO, MeZO-LoRA, LOZO and LOZO-M. First-order FT and FT-LoRA, as well as GaLore, serve only as upper-bound references.
  • On RoBERTa-large, the NLI tasks SNLI, MNLI and RTE were tested with k = 512 and k = 16 training examples. RoZO-M matches or exceeds full fine-tuning there.
  • The evaluation covers NLI, classification and extractive QA, using GLUE/SuperGLUE-style metrics (accuracy, F1, EM) on OPT models up to 66B. All methods use identical forward-query and FLOP budgets on A100 40GB GPUs.
  • The extra geometric operations cost O(r(m + n)), and peak memory is essentially the same as LOZO's.
  • The authors list three limitations: generative tasks remain untested, scaling beyond 100B parameters is still hard, and results are sensitive to the perturbation radius and the rank.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sungkyunkwan UniversitySuwon · South Korea9 works · 195 citations🎓Lanzhou UniversityLanzhou · China5 works · 24 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Zichen Song and Weijia Li. 2026. RoZO: Geometry-Aware Zeroth-Order Fine-Tuning on Low-Rank Adapters for Black-Box Large Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.80

@inproceedings{song2026rozo,
  title = {{RoZO: Geometry-Aware Zeroth-Order Fine-Tuning on Low-Rank Adapters for Black-Box Large Language Models}},
  author = {Zichen Song and Weijia Li},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.80},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.80}
}

References

43 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Shun-ichi Amari. 1993. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5):185–196. find
  3. Krishnakumar Balasubramanian and Saeed Ghadimi. 2018. Zeroth-order (non)-convex stochastic optimization via conditional gradient and gradient updates. Advances in Neural Information Processing Systems, 31. find
  4. Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. 2018. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754. — [arXiv:1812.04754](https://arxiv.org/abs/1812.04754) find
  5. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964. find
  6. Yongchang Hao, Yanshuai Cao, and Lili Mou. 2024. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293. — [arXiv:2402.03293](https://arxiv.org/abs/2402.03293) find
  7. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165.
  8. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  9. HanQin Cai, Daniel McKenzie, Wotao Yin, and Zhenliang Zhang. 2022. Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2):687–714. find
  10. Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. 2018. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137–2146. PMLR. find
  11. Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. 2023. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02025. — [arXiv:2310.02025](https://arxiv.org/abs/2310.02025) find
  12. Kaiyi Ji, Zhe Wang, Yi Zhou, and Yingbin Liang. 2019. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. In International conference on machine learning, pages 3100– 3109. PMLR. find
  13. Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. 2019. Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization. Advances in neural information processing systems, 32. find
  14. Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. 2025. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In The Thirteenth International Conference on Learning Representations. find
  15. Amit Dhurandhar, Tejaswini Pedapati, Avinash Balakrishnan, Pin-Yu Chen, Karthikeyan Shanmugam, and Ruchir Puri. 2019. Model agnostic contrastive explanations for structured data. arXiv preprint arXiv:1906.00117. — [arXiv:1906.00117](https://arxiv.org/abs/1906.00117) find
  16. John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. 2015. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806. find
  17. Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. 2024. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080. find
  18. Saeed Ghadimi and Guanghui Lan. 2013. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368. find
  19. Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  20. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  21. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838. find
  22. Zeman Li, Xinwei Zhang, and Meisam Razaviyayn. 2024. Addax: Memory-efficient fine-tuning of language models with a combination of forward-backward and forward-only passes. In 5th Workshop on practical ML for limited/low resource settings. find
  23. Xiangru Lian, Huan Zhang, Cho-Jui Hsieh, Yijun Huang, and Ji Liu. 2016. A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order. Advances in Neural Information Processing Systems, 29. find
  24. Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. 2019. signsgd via zeroth-order oracle. In International Conference on Learning Representations. find
  25. Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. 2018. Zeroth-order stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems, 31. find
  26. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075. find
  27. Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, and Virginia Smith. 2024. Grass: Compute efficient low-memory llm training with structured sparse gradients. arXiv preprint arXiv:2406.17660. find
  28. Yurii Nesterov and Vladimir Spokoiny. 2017. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566. find
  29. Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094. find
  30. Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017. Kvasir: A multi-class image dataset for computer aided gastroi… find
  31. Zhen Qu, Xian Tao, Mukesh Prasad, Fei Shen, Zhengtao Zhang, Xinyi Gong, and Guiguang Ding. 2024. Vcp-clip: A visual context prompting model for zero-shot anomaly segmentation. arXiv preprint arXiv:2407.12276. — [arXiv:2407.12276](https://arxiv.org/abs/2407.12276) find
  32. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8…
  33. Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. 2017. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454. — [arXiv:1706.04454](https://arxiv.org/abs/1706.04454) find
  34. James C Spall. 1992. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341. find
  35. Chun-Chen Tu, Paishun Ting, Pin-Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. 2019. Autozoom: Autoencoder-based zeroth order optimization method for attacking black-box neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pag… find
  36. Xiaoxing Wang, Wenxuan Guo, Jianlin Su, Xiaokang Yang, and Junchi Yan. 2022. Zarts: On zero-order optimization for neural architecture search. Advances in Neural Information Processing Systems, 35:12868– 12880. find
  37. Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, and 1 others. 2024. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592. — [arXiv:2402.11592](https://arxiv.org/abs/2402.11592) find
  38. Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jinfeng Yi, Mingyi Hong, Shiyu Chang, and Sijia Liu. 2022. How to robustify black-box ml models? a zeroth-order optimization perspective. arXiv preprint arXiv:2203.14195. — [arXiv:2203.14195](https://arxiv.org/abs/2203.14195) find
  39. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024a. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. find
  40. Pu Zhao, Sijia Liu, Pin-Yu Chen, Nghia Hoang, Kaidi Xu, Bhavya Kailkhura, and Xue Lin. 2019. On the design of black-box adversarial examples by leveraging gradient-free optimization and operator splitting method. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 121–… find
  41. Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. 2024b. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15173. find
  42. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, and 1 others. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
  43. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, and 1 others. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2026-rozo-geometry-aware-zeroth-order/