Fast Population-Based Reinforcement Learning on a Single Machine

Arthur Flajolet, Claire Bizon Monroc, Karim Beguir, Thomas Pierrot
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:6533-6547, 2022.

Abstract

Training populations of agents has demonstrated great promise in Reinforcement Learning for stabilizing training, improving exploration and asymptotic performance, and generating a diverse set of solutions. However, population-based training is often not considered by practitioners as it is perceived to be either prohibitively slow (when implemented sequentially), or computationally expensive (if agents are trained in parallel on independent accelerators). In this work, we compare implementations and revisit previous studies to show that the judicious use of compilation and vectorization allows population-based training to be performed on a single machine with one accelerator with minimal overhead compared to training a single agent. We also show that, when provided with a few accelerators, our protocols extend to large population sizes for applications such as hyperparameter tuning. We hope that this work and the public release of our code will encourage practitioners to use population-based learning techniques more frequently for their research and applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-flajolet22a, title = {Fast Population-Based Reinforcement Learning on a Single Machine}, author = {Flajolet, Arthur and Monroc, Claire Bizon and Beguir, Karim and Pierrot, Thomas}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {6533--6547}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/flajolet22a/flajolet22a.pdf}, url = {https://proceedings.mlr.press/v162/flajolet22a.html}, abstract = {Training populations of agents has demonstrated great promise in Reinforcement Learning for stabilizing training, improving exploration and asymptotic performance, and generating a diverse set of solutions. However, population-based training is often not considered by practitioners as it is perceived to be either prohibitively slow (when implemented sequentially), or computationally expensive (if agents are trained in parallel on independent accelerators). In this work, we compare implementations and revisit previous studies to show that the judicious use of compilation and vectorization allows population-based training to be performed on a single machine with one accelerator with minimal overhead compared to training a single agent. We also show that, when provided with a few accelerators, our protocols extend to large population sizes for applications such as hyperparameter tuning. We hope that this work and the public release of our code will encourage practitioners to use population-based learning techniques more frequently for their research and applications.} }
Endnote
%0 Conference Paper %T Fast Population-Based Reinforcement Learning on a Single Machine %A Arthur Flajolet %A Claire Bizon Monroc %A Karim Beguir %A Thomas Pierrot %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-flajolet22a %I PMLR %P 6533--6547 %U https://proceedings.mlr.press/v162/flajolet22a.html %V 162 %X Training populations of agents has demonstrated great promise in Reinforcement Learning for stabilizing training, improving exploration and asymptotic performance, and generating a diverse set of solutions. However, population-based training is often not considered by practitioners as it is perceived to be either prohibitively slow (when implemented sequentially), or computationally expensive (if agents are trained in parallel on independent accelerators). In this work, we compare implementations and revisit previous studies to show that the judicious use of compilation and vectorization allows population-based training to be performed on a single machine with one accelerator with minimal overhead compared to training a single agent. We also show that, when provided with a few accelerators, our protocols extend to large population sizes for applications such as hyperparameter tuning. We hope that this work and the public release of our code will encourage practitioners to use population-based learning techniques more frequently for their research and applications.
APA
Flajolet, A., Monroc, C.B., Beguir, K. & Pierrot, T.. (2022). Fast Population-Based Reinforcement Learning on a Single Machine. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:6533-6547 Available from https://proceedings.mlr.press/v162/flajolet22a.html.

Related Material