Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization

He Chen, Haochen Xu, Rujun Jiang, Anthony Man-Cho So
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:784-792, 2024.

Abstract

Hyperparameter tuning is an important task of machine learning, which can be formulated as a bilevel program (BLP). However, most existing algorithms are not applicable for BLP with non-smooth lower-level problems. To address this, we propose a single-level reformulation of the BLP based on lower-level duality without involving any implicit value function. To solve the reformulation, we propose a majorization minimization algorithm that marjorizes the constraint in each iteration. Furthermore, we show that the subproblems of the proposed algorithm for several widely-used hyperparameter turning models can be reformulated into conic programs that can be efficiently solved by the off-the-shelf solvers. We theoretically prove the convergence of the proposed algorithm and demonstrate its superiority through numerical experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-chen24a, title = {Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization}, author = {Chen, He and Xu, Haochen and Jiang, Rujun and Man-Cho So, Anthony}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {784--792}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/chen24a/chen24a.pdf}, url = {https://proceedings.mlr.press/v238/chen24a.html}, abstract = {Hyperparameter tuning is an important task of machine learning, which can be formulated as a bilevel program (BLP). However, most existing algorithms are not applicable for BLP with non-smooth lower-level problems. To address this, we propose a single-level reformulation of the BLP based on lower-level duality without involving any implicit value function. To solve the reformulation, we propose a majorization minimization algorithm that marjorizes the constraint in each iteration. Furthermore, we show that the subproblems of the proposed algorithm for several widely-used hyperparameter turning models can be reformulated into conic programs that can be efficiently solved by the off-the-shelf solvers. We theoretically prove the convergence of the proposed algorithm and demonstrate its superiority through numerical experiments.} }
Endnote
%0 Conference Paper %T Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization %A He Chen %A Haochen Xu %A Rujun Jiang %A Anthony Man-Cho So %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-chen24a %I PMLR %P 784--792 %U https://proceedings.mlr.press/v238/chen24a.html %V 238 %X Hyperparameter tuning is an important task of machine learning, which can be formulated as a bilevel program (BLP). However, most existing algorithms are not applicable for BLP with non-smooth lower-level problems. To address this, we propose a single-level reformulation of the BLP based on lower-level duality without involving any implicit value function. To solve the reformulation, we propose a majorization minimization algorithm that marjorizes the constraint in each iteration. Furthermore, we show that the subproblems of the proposed algorithm for several widely-used hyperparameter turning models can be reformulated into conic programs that can be efficiently solved by the off-the-shelf solvers. We theoretically prove the convergence of the proposed algorithm and demonstrate its superiority through numerical experiments.
APA
Chen, H., Xu, H., Jiang, R. & Man-Cho So, A.. (2024). Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:784-792 Available from https://proceedings.mlr.press/v238/chen24a.html.

Related Material