Hyperparameter optimization with approximate gradient

Fabian Pedregosa
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:737-746, 2016.

Abstract

Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-pedregosa16, title = {Hyperparameter optimization with approximate gradient}, author = {Pedregosa, Fabian}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {737--746}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/pedregosa16.pdf}, url = {https://proceedings.mlr.press/v48/pedregosa16.html}, abstract = {Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.} }
Endnote
%0 Conference Paper %T Hyperparameter optimization with approximate gradient %A Fabian Pedregosa %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-pedregosa16 %I PMLR %P 737--746 %U https://proceedings.mlr.press/v48/pedregosa16.html %V 48 %X Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.
RIS
TY - CPAPER TI - Hyperparameter optimization with approximate gradient AU - Fabian Pedregosa BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-pedregosa16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 737 EP - 746 L1 - http://proceedings.mlr.press/v48/pedregosa16.pdf UR - https://proceedings.mlr.press/v48/pedregosa16.html AB - Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods. ER -
APA
Pedregosa, F.. (2016). Hyperparameter optimization with approximate gradient. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:737-746 Available from https://proceedings.mlr.press/v48/pedregosa16.html.

Related Material