Automatic Goal Generation for Reinforcement Learning Agents

Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1515-1528, 2018.

Abstract

Reinforcement learning (RL) is a powerful technique to train an agent to perform a task; however, an agent that is trained using RL is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing in its environment. We use a generator network to propose tasks for the agent to try to accomplish, each task being specified as reaching a certain parametrized subset of the state-space. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent, thus automatically producing a curriculum. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment, even when only sparse rewards are available. Videos and code available at https://sites.google.com/view/goalgeneration4rl.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-florensa18a, title = {Automatic Goal Generation for Reinforcement Learning Agents}, author = {Florensa, Carlos and Held, David and Geng, Xinyang and Abbeel, Pieter}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {1515--1528}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/florensa18a/florensa18a.pdf}, url = {https://proceedings.mlr.press/v80/florensa18a.html}, abstract = {Reinforcement learning (RL) is a powerful technique to train an agent to perform a task; however, an agent that is trained using RL is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing in its environment. We use a generator network to propose tasks for the agent to try to accomplish, each task being specified as reaching a certain parametrized subset of the state-space. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent, thus automatically producing a curriculum. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment, even when only sparse rewards are available. Videos and code available at https://sites.google.com/view/goalgeneration4rl.} }
Endnote
%0 Conference Paper %T Automatic Goal Generation for Reinforcement Learning Agents %A Carlos Florensa %A David Held %A Xinyang Geng %A Pieter Abbeel %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-florensa18a %I PMLR %P 1515--1528 %U https://proceedings.mlr.press/v80/florensa18a.html %V 80 %X Reinforcement learning (RL) is a powerful technique to train an agent to perform a task; however, an agent that is trained using RL is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing in its environment. We use a generator network to propose tasks for the agent to try to accomplish, each task being specified as reaching a certain parametrized subset of the state-space. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent, thus automatically producing a curriculum. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment, even when only sparse rewards are available. Videos and code available at https://sites.google.com/view/goalgeneration4rl.
APA
Florensa, C., Held, D., Geng, X. & Abbeel, P.. (2018). Automatic Goal Generation for Reinforcement Learning Agents. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:1515-1528 Available from https://proceedings.mlr.press/v80/florensa18a.html.

Related Material