MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters

Arsalan Sharifnassab, Saber Salehkaleybar, Richard S. Sutton
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:54300-54325, 2025.

Abstract

We address the challenge of optimizing meta-parameters (hyperparameters) in machine learning, a key factor for efficient training and high model performance. Rather than relying on expensive meta-parameter search methods, we introduce MetaOptimize: a dynamic approach that adjusts meta-parameters, particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes on the fly to minimize a specific form of regret that considers the long-term impact of step sizes on training, through a discounted sum of future losses. We also introduce lower-complexity variants of MetaOptimize that, in conjunction with its adaptability to various optimization algorithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-sharifnassab25a, title = {{M}eta{O}ptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters}, author = {Sharifnassab, Arsalan and Salehkaleybar, Saber and Sutton, Richard S.}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {54300--54325}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/sharifnassab25a/sharifnassab25a.pdf}, url = {https://proceedings.mlr.press/v267/sharifnassab25a.html}, abstract = {We address the challenge of optimizing meta-parameters (hyperparameters) in machine learning, a key factor for efficient training and high model performance. Rather than relying on expensive meta-parameter search methods, we introduce MetaOptimize: a dynamic approach that adjusts meta-parameters, particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes on the fly to minimize a specific form of regret that considers the long-term impact of step sizes on training, through a discounted sum of future losses. We also introduce lower-complexity variants of MetaOptimize that, in conjunction with its adaptability to various optimization algorithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.} }
Endnote
%0 Conference Paper %T MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters %A Arsalan Sharifnassab %A Saber Salehkaleybar %A Richard S. Sutton %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-sharifnassab25a %I PMLR %P 54300--54325 %U https://proceedings.mlr.press/v267/sharifnassab25a.html %V 267 %X We address the challenge of optimizing meta-parameters (hyperparameters) in machine learning, a key factor for efficient training and high model performance. Rather than relying on expensive meta-parameter search methods, we introduce MetaOptimize: a dynamic approach that adjusts meta-parameters, particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes on the fly to minimize a specific form of regret that considers the long-term impact of step sizes on training, through a discounted sum of future losses. We also introduce lower-complexity variants of MetaOptimize that, in conjunction with its adaptability to various optimization algorithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.
APA
Sharifnassab, A., Salehkaleybar, S. & Sutton, R.S.. (2025). MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:54300-54325 Available from https://proceedings.mlr.press/v267/sharifnassab25a.html.

Related Material