ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models

Lejs Deen Behric, Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2170-2178, 2026.

Abstract

Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large language models (LLMs) because it eliminates the memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural language tasks, ConMeZO is up to 2$\times$ faster than MeZO while retaining the low-memory footprint of zeroth-order methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-behric26a, title = { ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models }, author = {Behric, Lejs Deen and Zhang, Liang and Li, Bingcong and Thekumparampil, Kiran Koshy}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2170--2178}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/behric26a/behric26a.pdf}, url = {https://proceedings.mlr.press/v300/behric26a.html}, abstract = { Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large language models (LLMs) because it eliminates the memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural language tasks, ConMeZO is up to 2$\times$ faster than MeZO while retaining the low-memory footprint of zeroth-order methods. } }
Endnote
%0 Conference Paper %T ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models %A Lejs Deen Behric %A Liang Zhang %A Bingcong Li %A Kiran Koshy Thekumparampil %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-behric26a %I PMLR %P 2170--2178 %U https://proceedings.mlr.press/v300/behric26a.html %V 300 %X Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large language models (LLMs) because it eliminates the memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural language tasks, ConMeZO is up to 2$\times$ faster than MeZO while retaining the low-memory footprint of zeroth-order methods.
APA
Behric, L.D., Zhang, L., Li, B. & Thekumparampil, K.K.. (2026). ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2170-2178 Available from https://proceedings.mlr.press/v300/behric26a.html.

Related Material