Learning Action Representations for Reinforcement Learning

Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, Philip Thomas
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:941-950, 2019.

Abstract

Most model-free reinforcement learning methods leverage state representations (embeddings) for generalization, but either ignore structure in the space of actions or assume the structure is provided a priori. We show how a policy can be decomposed into a component that acts in a low-dimensional space of action representations and a component that transforms these representations into actual actions. These representations improve generalization over large, finite action sets by allowing the agent to infer the outcomes of actions similar to actions already taken. We provide an algorithm to both learn and use action representations and provide conditions for its convergence. The efficacy of the proposed method is demonstrated on large-scale real-world problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-chandak19a, title = {Learning Action Representations for Reinforcement Learning}, author = {Chandak, Yash and Theocharous, Georgios and Kostas, James and Jordan, Scott and Thomas, Philip}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {941--950}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/chandak19a/chandak19a.pdf}, url = {https://proceedings.mlr.press/v97/chandak19a.html}, abstract = {Most model-free reinforcement learning methods leverage state representations (embeddings) for generalization, but either ignore structure in the space of actions or assume the structure is provided a priori. We show how a policy can be decomposed into a component that acts in a low-dimensional space of action representations and a component that transforms these representations into actual actions. These representations improve generalization over large, finite action sets by allowing the agent to infer the outcomes of actions similar to actions already taken. We provide an algorithm to both learn and use action representations and provide conditions for its convergence. The efficacy of the proposed method is demonstrated on large-scale real-world problems.} }
Endnote
%0 Conference Paper %T Learning Action Representations for Reinforcement Learning %A Yash Chandak %A Georgios Theocharous %A James Kostas %A Scott Jordan %A Philip Thomas %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-chandak19a %I PMLR %P 941--950 %U https://proceedings.mlr.press/v97/chandak19a.html %V 97 %X Most model-free reinforcement learning methods leverage state representations (embeddings) for generalization, but either ignore structure in the space of actions or assume the structure is provided a priori. We show how a policy can be decomposed into a component that acts in a low-dimensional space of action representations and a component that transforms these representations into actual actions. These representations improve generalization over large, finite action sets by allowing the agent to infer the outcomes of actions similar to actions already taken. We provide an algorithm to both learn and use action representations and provide conditions for its convergence. The efficacy of the proposed method is demonstrated on large-scale real-world problems.
APA
Chandak, Y., Theocharous, G., Kostas, J., Jordan, S. & Thomas, P.. (2019). Learning Action Representations for Reinforcement Learning. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:941-950 Available from https://proceedings.mlr.press/v97/chandak19a.html.

Related Material