GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning

Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, Dieter Fox
Proceedings of The 2nd Conference on Robot Learning, PMLR 87:270-282, 2018.

Abstract

Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train more challenging locomotion tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v87-liang18a, title = {GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning}, author = {Liang, Jacky and Makoviychuk, Viktor and Handa, Ankur and Chentanez, Nuttapong and Macklin, Miles and Fox, Dieter}, booktitle = {Proceedings of The 2nd Conference on Robot Learning}, pages = {270--282}, year = {2018}, editor = {Billard, Aude and Dragan, Anca and Peters, Jan and Morimoto, Jun}, volume = {87}, series = {Proceedings of Machine Learning Research}, month = {29--31 Oct}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v87/liang18a/liang18a.pdf}, url = {https://proceedings.mlr.press/v87/liang18a.html}, abstract = {Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train more challenging locomotion tasks. } }
Endnote
%0 Conference Paper %T GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning %A Jacky Liang %A Viktor Makoviychuk %A Ankur Handa %A Nuttapong Chentanez %A Miles Macklin %A Dieter Fox %B Proceedings of The 2nd Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2018 %E Aude Billard %E Anca Dragan %E Jan Peters %E Jun Morimoto %F pmlr-v87-liang18a %I PMLR %P 270--282 %U https://proceedings.mlr.press/v87/liang18a.html %V 87 %X Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train more challenging locomotion tasks.
APA
Liang, J., Makoviychuk, V., Handa, A., Chentanez, N., Macklin, M. & Fox, D.. (2018). GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning. Proceedings of The 2nd Conference on Robot Learning, in Proceedings of Machine Learning Research 87:270-282 Available from https://proceedings.mlr.press/v87/liang18a.html.

Related Material