Optimizing Millions of Hyperparameters by Implicit Differentiation
[edit]
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:15401552, 2020.
Abstract
We propose an algorithm for inexpensive gradientbased hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyperparameters. For example, we learn a dataaugmentation network—where every weight is a hyperparameter tuned for validation performance—outputting augmented training examples. Jointly tuning weights and hyperparameters is only a few times more costly in memory and compute than standard training.
Related Material


