Option Discovery in the Absence of Rewards with Manifold Analysis

Amitay Bar, Ronen Talmon, Ron Meir
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:664-674, 2020.

Abstract

Options have been shown to be an effective tool in reinforcement learning, facilitating improved exploration and learning. In this paper, we present an approach based on spectral graph theory and derive an algorithm that systematically discovers options without access to a specific reward or task assignment. As opposed to the common practice used in previous methods, our algorithm makes full use of the spectrum of the graph Laplacian. Incorporating modes associated with higher graph frequencies unravels domain subtleties, which are shown to be useful for option discovery. Using geometric and manifold-based analysis, we present a theoretical justification for the algorithm. In addition, we showcase its performance in several domains, demonstrating clear improvements compared to competing methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-bar20a, title = {Option Discovery in the Absence of Rewards with Manifold Analysis}, author = {Bar, Amitay and Talmon, Ronen and Meir, Ron}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {664--674}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/bar20a/bar20a.pdf}, url = {https://proceedings.mlr.press/v119/bar20a.html}, abstract = {Options have been shown to be an effective tool in reinforcement learning, facilitating improved exploration and learning. In this paper, we present an approach based on spectral graph theory and derive an algorithm that systematically discovers options without access to a specific reward or task assignment. As opposed to the common practice used in previous methods, our algorithm makes full use of the spectrum of the graph Laplacian. Incorporating modes associated with higher graph frequencies unravels domain subtleties, which are shown to be useful for option discovery. Using geometric and manifold-based analysis, we present a theoretical justification for the algorithm. In addition, we showcase its performance in several domains, demonstrating clear improvements compared to competing methods.} }
Endnote
%0 Conference Paper %T Option Discovery in the Absence of Rewards with Manifold Analysis %A Amitay Bar %A Ronen Talmon %A Ron Meir %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-bar20a %I PMLR %P 664--674 %U https://proceedings.mlr.press/v119/bar20a.html %V 119 %X Options have been shown to be an effective tool in reinforcement learning, facilitating improved exploration and learning. In this paper, we present an approach based on spectral graph theory and derive an algorithm that systematically discovers options without access to a specific reward or task assignment. As opposed to the common practice used in previous methods, our algorithm makes full use of the spectrum of the graph Laplacian. Incorporating modes associated with higher graph frequencies unravels domain subtleties, which are shown to be useful for option discovery. Using geometric and manifold-based analysis, we present a theoretical justification for the algorithm. In addition, we showcase its performance in several domains, demonstrating clear improvements compared to competing methods.
APA
Bar, A., Talmon, R. & Meir, R.. (2020). Option Discovery in the Absence of Rewards with Manifold Analysis. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:664-674 Available from https://proceedings.mlr.press/v119/bar20a.html.

Related Material