Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent

Suriya Gunasekar, Blake Woodworth, Nathan Srebro
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:2305-2313, 2021.

Abstract

We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a “full” forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in Rd, even when the metric tensor is not a Hessian, and thus there is no “dual.”

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-gunasekar21a, title = { Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent }, author = {Gunasekar, Suriya and Woodworth, Blake and Srebro, Nathan}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {2305--2313}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/gunasekar21a/gunasekar21a.pdf}, url = {https://proceedings.mlr.press/v130/gunasekar21a.html}, abstract = { We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a “full” forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in $\mathbb{R}^d$, even when the metric tensor is not a Hessian, and thus there is no “dual.” } }
Endnote
%0 Conference Paper %T Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent %A Suriya Gunasekar %A Blake Woodworth %A Nathan Srebro %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-gunasekar21a %I PMLR %P 2305--2313 %U https://proceedings.mlr.press/v130/gunasekar21a.html %V 130 %X We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a “full” forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in $\mathbb{R}^d$, even when the metric tensor is not a Hessian, and thus there is no “dual.”
APA
Gunasekar, S., Woodworth, B. & Srebro, N.. (2021). Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:2305-2313 Available from https://proceedings.mlr.press/v130/gunasekar21a.html.

Related Material