Learning to Explore via Meta-Policy Gradient

Tianbing Xu, Qiang Liu, Liang Zhao, Jian Peng
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5463-5472, 2018.

Abstract

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore local regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a global exploration that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning continuous control tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-xu18d, title = {Learning to Explore via Meta-Policy Gradient}, author = {Xu, Tianbing and Liu, Qiang and Zhao, Liang and Peng, Jian}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {5463--5472}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/xu18d/xu18d.pdf}, url = {https://proceedings.mlr.press/v80/xu18d.html}, abstract = {The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore local regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a global exploration that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning continuous control tasks.} }
Endnote
%0 Conference Paper %T Learning to Explore via Meta-Policy Gradient %A Tianbing Xu %A Qiang Liu %A Liang Zhao %A Jian Peng %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-xu18d %I PMLR %P 5463--5472 %U https://proceedings.mlr.press/v80/xu18d.html %V 80 %X The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore local regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a global exploration that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning continuous control tasks.
APA
Xu, T., Liu, Q., Zhao, L. & Peng, J.. (2018). Learning to Explore via Meta-Policy Gradient. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:5463-5472 Available from https://proceedings.mlr.press/v80/xu18d.html.

Related Material