Representation Learning on Graphs: A Reinforcement Learning Application

Sephora Madjiheurem, Laura Toni
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:3391-3399, 2019.

Abstract

In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low dimensions and we highlight the importance of features learning for an improved low-dimensional value function approximation. Then, we adopt different representation learning algorithms on graphs to learn the basis functions that best represent the value function. We empirically show that node2vec, an algorithm for scalable feature learning in networks, and Graph Auto-Encoder constantly outperform the commonly used smooth proto-value functions in low-dimensional feature space.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-madjiheurem19a, title = {Representation Learning on Graphs: A Reinforcement Learning Application}, author = {Madjiheurem, Sephora and Toni, Laura}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {3391--3399}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/madjiheurem19a/madjiheurem19a.pdf}, url = {https://proceedings.mlr.press/v89/madjiheurem19a.html}, abstract = {In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low dimensions and we highlight the importance of features learning for an improved low-dimensional value function approximation. Then, we adopt different representation learning algorithms on graphs to learn the basis functions that best represent the value function. We empirically show that node2vec, an algorithm for scalable feature learning in networks, and Graph Auto-Encoder constantly outperform the commonly used smooth proto-value functions in low-dimensional feature space.} }
Endnote
%0 Conference Paper %T Representation Learning on Graphs: A Reinforcement Learning Application %A Sephora Madjiheurem %A Laura Toni %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-madjiheurem19a %I PMLR %P 3391--3399 %U https://proceedings.mlr.press/v89/madjiheurem19a.html %V 89 %X In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low dimensions and we highlight the importance of features learning for an improved low-dimensional value function approximation. Then, we adopt different representation learning algorithms on graphs to learn the basis functions that best represent the value function. We empirically show that node2vec, an algorithm for scalable feature learning in networks, and Graph Auto-Encoder constantly outperform the commonly used smooth proto-value functions in low-dimensional feature space.
APA
Madjiheurem, S. & Toni, L.. (2019). Representation Learning on Graphs: A Reinforcement Learning Application. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:3391-3399 Available from https://proceedings.mlr.press/v89/madjiheurem19a.html.

Related Material