Optimizing Millions of Hyperparameters by Implicit Differentiation

Jonathan Lorraine, Paul Vicol, David Duvenaud
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:1540-1552, 2020.

Abstract

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network—where every weight is a hyperparameter tuned for validation performance—outputting augmented training examples. Jointly tuning weights and hyper-parameters is only a few times more costly in memory and compute than standard training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-lorraine20a, title = {Optimizing Millions of Hyperparameters by Implicit Differentiation}, author = {Lorraine, Jonathan and Vicol, Paul and Duvenaud, David}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {1540--1552}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/lorraine20a/lorraine20a.pdf}, url = {https://proceedings.mlr.press/v108/lorraine20a.html}, abstract = {We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network—where every weight is a hyperparameter tuned for validation performance—outputting augmented training examples. Jointly tuning weights and hyper-parameters is only a few times more costly in memory and compute than standard training.} }
Endnote
%0 Conference Paper %T Optimizing Millions of Hyperparameters by Implicit Differentiation %A Jonathan Lorraine %A Paul Vicol %A David Duvenaud %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-lorraine20a %I PMLR %P 1540--1552 %U https://proceedings.mlr.press/v108/lorraine20a.html %V 108 %X We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network—where every weight is a hyperparameter tuned for validation performance—outputting augmented training examples. Jointly tuning weights and hyper-parameters is only a few times more costly in memory and compute than standard training.
APA
Lorraine, J., Vicol, P. & Duvenaud, D.. (2020). Optimizing Millions of Hyperparameters by Implicit Differentiation. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:1540-1552 Available from https://proceedings.mlr.press/v108/lorraine20a.html.

Related Material