The State of Sparse Training in Deep Reinforcement Learning

Laura Graesser, Utku Evci, Erich Elsen, Pablo Samuel Castro
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:7766-7792, 2022.

Abstract

The use of sparse neural networks has seen rapid growth in recent years, particularly in computer vision. Their appeal stems largely from the reduced number of parameters required to train and store, as well as in an increase in learning efficiency. Somewhat surprisingly, there have been very few efforts exploring their use in Deep Reinforcement Learning (DRL). In this work we perform a systematic investigation into applying a number of existing sparse training techniques on a variety of DRL agents and environments. Our results corroborate the findings from sparse training in the computer vision domain {–}sparse networks perform better than dense networks for the same parameter count{–} in the DRL domain. We provide detailed analyses on how the various components in DRL are affected by the use of sparse networks and conclude by suggesting promising avenues for improving the effectiveness of sparse training methods, as well as for advancing their use in DRL.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-graesser22a, title = {The State of Sparse Training in Deep Reinforcement Learning}, author = {Graesser, Laura and Evci, Utku and Elsen, Erich and Castro, Pablo Samuel}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {7766--7792}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/graesser22a/graesser22a.pdf}, url = {https://proceedings.mlr.press/v162/graesser22a.html}, abstract = {The use of sparse neural networks has seen rapid growth in recent years, particularly in computer vision. Their appeal stems largely from the reduced number of parameters required to train and store, as well as in an increase in learning efficiency. Somewhat surprisingly, there have been very few efforts exploring their use in Deep Reinforcement Learning (DRL). In this work we perform a systematic investigation into applying a number of existing sparse training techniques on a variety of DRL agents and environments. Our results corroborate the findings from sparse training in the computer vision domain {–}sparse networks perform better than dense networks for the same parameter count{–} in the DRL domain. We provide detailed analyses on how the various components in DRL are affected by the use of sparse networks and conclude by suggesting promising avenues for improving the effectiveness of sparse training methods, as well as for advancing their use in DRL.} }
Endnote
%0 Conference Paper %T The State of Sparse Training in Deep Reinforcement Learning %A Laura Graesser %A Utku Evci %A Erich Elsen %A Pablo Samuel Castro %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-graesser22a %I PMLR %P 7766--7792 %U https://proceedings.mlr.press/v162/graesser22a.html %V 162 %X The use of sparse neural networks has seen rapid growth in recent years, particularly in computer vision. Their appeal stems largely from the reduced number of parameters required to train and store, as well as in an increase in learning efficiency. Somewhat surprisingly, there have been very few efforts exploring their use in Deep Reinforcement Learning (DRL). In this work we perform a systematic investigation into applying a number of existing sparse training techniques on a variety of DRL agents and environments. Our results corroborate the findings from sparse training in the computer vision domain {–}sparse networks perform better than dense networks for the same parameter count{–} in the DRL domain. We provide detailed analyses on how the various components in DRL are affected by the use of sparse networks and conclude by suggesting promising avenues for improving the effectiveness of sparse training methods, as well as for advancing their use in DRL.
APA
Graesser, L., Evci, U., Elsen, E. & Castro, P.S.. (2022). The State of Sparse Training in Deep Reinforcement Learning. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:7766-7792 Available from https://proceedings.mlr.press/v162/graesser22a.html.

Related Material