Automatic Differentiation of Some FirstOrder Methods in Parametric Optimization
[edit]
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:15841594, 2020.
Abstract
We aim at computing the derivative of the solution to a parametric optimization problem with respect to the involved parameters. For a class broader than that of strongly convex functions, this can be achieved by automatic differentiation of iterative minimization algorithms. If the iterative algorithm converges pointwise, then we prove that the derivative sequence also converges pointwise to the derivative of the minimizer with respect to the parameters. Moreover, we provide convergence rates for both sequences. In particular, we prove that the accelerated convergence rate of the Heavyball method compared to Gradient Descent also accelerates the derivative computation. An experiment with L2Regularized Logistic Regression validates the theoretical results.
Related Material


