Phasic Policy Gradient

Karl W Cobbe, Jacob Hilton, Oleg Klimov, John Schulman
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:2020-2027, 2021.

Abstract

We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-cobbe21a, title = {Phasic Policy Gradient}, author = {Cobbe, Karl W and Hilton, Jacob and Klimov, Oleg and Schulman, John}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {2020--2027}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/cobbe21a/cobbe21a.pdf}, url = {https://proceedings.mlr.press/v139/cobbe21a.html}, abstract = {We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.} }
Endnote
%0 Conference Paper %T Phasic Policy Gradient %A Karl W Cobbe %A Jacob Hilton %A Oleg Klimov %A John Schulman %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-cobbe21a %I PMLR %P 2020--2027 %U https://proceedings.mlr.press/v139/cobbe21a.html %V 139 %X We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.
APA
Cobbe, K.W., Hilton, J., Klimov, O. & Schulman, J.. (2021). Phasic Policy Gradient. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:2020-2027 Available from https://proceedings.mlr.press/v139/cobbe21a.html.

Related Material