Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing

Kaixin Wang, Kuangqi Zhou, Qixin Zhang, Jie Shao, Bryan Hooi, Jiashi Feng
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:11003-11012, 2021.

Abstract

The Laplacian representation recently gains increasing attention for reinforcement learning as it provides succinct and informative representation for states, by taking the eigenvectors of the Laplacian matrix of the state-transition graph as state embeddings. Such representation captures the geometry of the underlying state space and is beneficial to RL tasks such as option discovery and reward shaping. To approximate the Laplacian representation in large (or even continuous) state spaces, recent works propose to minimize a spectral graph drawing objective, which however has infinitely many global minimizers other than the eigenvectors. As a result, their learned Laplacian representation may differ from the ground truth. To solve this problem, we reformulate the graph drawing objective into a generalized form and derive a new learning objective, which is proved to have eigenvectors as its unique global minimizer. It enables learning high-quality Laplacian representations that faithfully approximate the ground truth. We validate this via comprehensive experiments on a set of gridworld and continuous control environments. Moreover, we show that our learned Laplacian representations lead to more exploratory options and better reward shaping.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-wang21ae, title = {Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing}, author = {Wang, Kaixin and Zhou, Kuangqi and Zhang, Qixin and Shao, Jie and Hooi, Bryan and Feng, Jiashi}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {11003--11012}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/wang21ae/wang21ae.pdf}, url = {https://proceedings.mlr.press/v139/wang21ae.html}, abstract = {The Laplacian representation recently gains increasing attention for reinforcement learning as it provides succinct and informative representation for states, by taking the eigenvectors of the Laplacian matrix of the state-transition graph as state embeddings. Such representation captures the geometry of the underlying state space and is beneficial to RL tasks such as option discovery and reward shaping. To approximate the Laplacian representation in large (or even continuous) state spaces, recent works propose to minimize a spectral graph drawing objective, which however has infinitely many global minimizers other than the eigenvectors. As a result, their learned Laplacian representation may differ from the ground truth. To solve this problem, we reformulate the graph drawing objective into a generalized form and derive a new learning objective, which is proved to have eigenvectors as its unique global minimizer. It enables learning high-quality Laplacian representations that faithfully approximate the ground truth. We validate this via comprehensive experiments on a set of gridworld and continuous control environments. Moreover, we show that our learned Laplacian representations lead to more exploratory options and better reward shaping.} }
Endnote
%0 Conference Paper %T Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing %A Kaixin Wang %A Kuangqi Zhou %A Qixin Zhang %A Jie Shao %A Bryan Hooi %A Jiashi Feng %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-wang21ae %I PMLR %P 11003--11012 %U https://proceedings.mlr.press/v139/wang21ae.html %V 139 %X The Laplacian representation recently gains increasing attention for reinforcement learning as it provides succinct and informative representation for states, by taking the eigenvectors of the Laplacian matrix of the state-transition graph as state embeddings. Such representation captures the geometry of the underlying state space and is beneficial to RL tasks such as option discovery and reward shaping. To approximate the Laplacian representation in large (or even continuous) state spaces, recent works propose to minimize a spectral graph drawing objective, which however has infinitely many global minimizers other than the eigenvectors. As a result, their learned Laplacian representation may differ from the ground truth. To solve this problem, we reformulate the graph drawing objective into a generalized form and derive a new learning objective, which is proved to have eigenvectors as its unique global minimizer. It enables learning high-quality Laplacian representations that faithfully approximate the ground truth. We validate this via comprehensive experiments on a set of gridworld and continuous control environments. Moreover, we show that our learned Laplacian representations lead to more exploratory options and better reward shaping.
APA
Wang, K., Zhou, K., Zhang, Q., Shao, J., Hooi, B. & Feng, J.. (2021). Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:11003-11012 Available from https://proceedings.mlr.press/v139/wang21ae.html.

Related Material