Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate

Huangyu Xu, Jingqin Yang, Qianqian Xu, Jiaye Teng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:142465-142496, 2026.

Abstract

Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is Lp regularization. However, it may encounter optimization instability due to the unbounded gradients when 0$<$p$<$1. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to lp-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the L1-regularization approach while preserving test accuracy.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-xu26bx, title = {Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate}, author = {Xu, Huangyu and Yang, Jingqin and Xu, Qianqian and Teng, Jiaye}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {142465--142496}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xu26bx/xu26bx.pdf}, url = {https://proceedings.mlr.press/v306/xu26bx.html}, abstract = {Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is Lp regularization. However, it may encounter optimization instability due to the unbounded gradients when 0$<$p$<$1. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to lp-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the L1-regularization approach while preserving test accuracy.} }
Endnote
%0 Conference Paper %T Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate %A Huangyu Xu %A Jingqin Yang %A Qianqian Xu %A Jiaye Teng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xu26bx %I PMLR %P 142465--142496 %U https://proceedings.mlr.press/v306/xu26bx.html %V 306 %X Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is Lp regularization. However, it may encounter optimization instability due to the unbounded gradients when 0$<$p$<$1. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to lp-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the L1-regularization approach while preserving test accuracy.
APA
Xu, H., Yang, J., Xu, Q. & Teng, J.. (2026). Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:142465-142496 Available from https://proceedings.mlr.press/v306/xu26bx.html.

Related Material