Stochastic Flows and Geometric Optimization on the Orthogonal Group

Krzysztof Choromanski, David Cheikhi, Jared Davis, Valerii Likhosherstov, Achille Nazaret, Achraf Bahamou, Xingyou Song, Mrugank Akarte, Jack Parker-Holder, Jacob Bergquist, Yuan Gao, Aldo Pacchiano, Tamas Sarlos, Adrian Weller, Vikas Sindhwani
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1918-1928, 2020.

Abstract

We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group O(d) and naturally reductive homogeneous manifolds obtained from the action of the rotation group SO(d). We theoretically and experimentally demonstrate that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, normalizing flows and metric learning. We show an intriguing connection between efficient stochastic optimization on the orthogonal group and graph theory (e.g. matching problem, partition functions over graphs, graph-coloring). We leverage the theory of Lie groups and provide theoretical results for the designed class of algorithms. We demonstrate broad applicability of our methods by showing strong performance on the seemingly unrelated tasks of learning world models to obtain stable policies for the most difficult Humanoid agent from OpenAI Gym and improving convolutional neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-choromanski20a, title = {Stochastic Flows and Geometric Optimization on the Orthogonal Group}, author = {Choromanski, Krzysztof and Cheikhi, David and Davis, Jared and Likhosherstov, Valerii and Nazaret, Achille and Bahamou, Achraf and Song, Xingyou and Akarte, Mrugank and Parker-Holder, Jack and Bergquist, Jacob and Gao, Yuan and Pacchiano, Aldo and Sarlos, Tamas and Weller, Adrian and Sindhwani, Vikas}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {1918--1928}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/choromanski20a/choromanski20a.pdf}, url = {https://proceedings.mlr.press/v119/choromanski20a.html}, abstract = {We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group O(d) and naturally reductive homogeneous manifolds obtained from the action of the rotation group SO(d). We theoretically and experimentally demonstrate that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, normalizing flows and metric learning. We show an intriguing connection between efficient stochastic optimization on the orthogonal group and graph theory (e.g. matching problem, partition functions over graphs, graph-coloring). We leverage the theory of Lie groups and provide theoretical results for the designed class of algorithms. We demonstrate broad applicability of our methods by showing strong performance on the seemingly unrelated tasks of learning world models to obtain stable policies for the most difficult Humanoid agent from OpenAI Gym and improving convolutional neural networks.} }
Endnote
%0 Conference Paper %T Stochastic Flows and Geometric Optimization on the Orthogonal Group %A Krzysztof Choromanski %A David Cheikhi %A Jared Davis %A Valerii Likhosherstov %A Achille Nazaret %A Achraf Bahamou %A Xingyou Song %A Mrugank Akarte %A Jack Parker-Holder %A Jacob Bergquist %A Yuan Gao %A Aldo Pacchiano %A Tamas Sarlos %A Adrian Weller %A Vikas Sindhwani %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-choromanski20a %I PMLR %P 1918--1928 %U https://proceedings.mlr.press/v119/choromanski20a.html %V 119 %X We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group O(d) and naturally reductive homogeneous manifolds obtained from the action of the rotation group SO(d). We theoretically and experimentally demonstrate that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, normalizing flows and metric learning. We show an intriguing connection between efficient stochastic optimization on the orthogonal group and graph theory (e.g. matching problem, partition functions over graphs, graph-coloring). We leverage the theory of Lie groups and provide theoretical results for the designed class of algorithms. We demonstrate broad applicability of our methods by showing strong performance on the seemingly unrelated tasks of learning world models to obtain stable policies for the most difficult Humanoid agent from OpenAI Gym and improving convolutional neural networks.
APA
Choromanski, K., Cheikhi, D., Davis, J., Likhosherstov, V., Nazaret, A., Bahamou, A., Song, X., Akarte, M., Parker-Holder, J., Bergquist, J., Gao, Y., Pacchiano, A., Sarlos, T., Weller, A. & Sindhwani, V.. (2020). Stochastic Flows and Geometric Optimization on the Orthogonal Group. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:1918-1928 Available from https://proceedings.mlr.press/v119/choromanski20a.html.

Related Material