Differentiable Architecture Search for Reinforcement Learning

Yingjie Miao, Xingyou Song, John D Co-Reyes, Daiyi Peng, Summer Yue, Eugene Brevdo, Aleksandra Faust
Proceedings of the First International Conference on Automated Machine Learning, PMLR 188:20/1-17, 2022.

Abstract

In this paper, we investigate the fundamental question: To what extent are gradient-based neural architecture search (NAS) techniques applicable to RL? Using the original DARTS as a convenient baseline, we discover that the discrete architectures found can achieve up to 250% performance compared to manual architecture designs on both discrete and continuous action space environments across off-policy and on-policy RL algorithms, at only 3{\texttimes} more computation time. Furthermore, through numerous ablation studies, we systematically verify that not only does DARTS correctly upweight operations during its supernet phrase, but also gradually improves resulting discrete cells up to 30{\texttimes} more efficiently than random search, suggesting DARTS is surprisingly an effective tool for improving architectures in RL.

Cite this Paper

BibTeX
@InProceedings{pmlr-v188-miao22a, title = {Differentiable Architecture Search for Reinforcement Learning}, author = {Miao, Yingjie and Song, Xingyou and Co-Reyes, John D and Peng, Daiyi and Yue, Summer and Brevdo, Eugene and Faust, Aleksandra}, booktitle = {Proceedings of the First International Conference on Automated Machine Learning}, pages = {20/1--17}, year = {2022}, editor = {Guyon, Isabelle and Lindauer, Marius and van der Schaar, Mihaela and Hutter, Frank and Garnett, Roman}, volume = {188}, series = {Proceedings of Machine Learning Research}, month = {25--27 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v188/miao22a/miao22a.pdf}, url = {https://proceedings.mlr.press/v188/miao22a.html}, abstract = {In this paper, we investigate the fundamental question: To what extent are gradient-based neural architecture search (NAS) techniques applicable to RL? Using the original DARTS as a convenient baseline, we discover that the discrete architectures found can achieve up to 250% performance compared to manual architecture designs on both discrete and continuous action space environments across off-policy and on-policy RL algorithms, at only 3{\texttimes} more computation time. Furthermore, through numerous ablation studies, we systematically verify that not only does DARTS correctly upweight operations during its supernet phrase, but also gradually improves resulting discrete cells up to 30{\texttimes} more efficiently than random search, suggesting DARTS is surprisingly an effective tool for improving architectures in RL.} }
Endnote
%0 Conference Paper %T Differentiable Architecture Search for Reinforcement Learning %A Yingjie Miao %A Xingyou Song %A John D Co-Reyes %A Daiyi Peng %A Summer Yue %A Eugene Brevdo %A Aleksandra Faust %B Proceedings of the First International Conference on Automated Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Isabelle Guyon %E Marius Lindauer %E Mihaela van der Schaar %E Frank Hutter %E Roman Garnett %F pmlr-v188-miao22a %I PMLR %P 20/1--17 %U https://proceedings.mlr.press/v188/miao22a.html %V 188 %X In this paper, we investigate the fundamental question: To what extent are gradient-based neural architecture search (NAS) techniques applicable to RL? Using the original DARTS as a convenient baseline, we discover that the discrete architectures found can achieve up to 250% performance compared to manual architecture designs on both discrete and continuous action space environments across off-policy and on-policy RL algorithms, at only 3{\texttimes} more computation time. Furthermore, through numerous ablation studies, we systematically verify that not only does DARTS correctly upweight operations during its supernet phrase, but also gradually improves resulting discrete cells up to 30{\texttimes} more efficiently than random search, suggesting DARTS is surprisingly an effective tool for improving architectures in RL.
APA
Miao, Y., Song, X., Co-Reyes, J.D., Peng, D., Yue, S., Brevdo, E. & Faust, A.. (2022). Differentiable Architecture Search for Reinforcement Learning. Proceedings of the First International Conference on Automated Machine Learning, in Proceedings of Machine Learning Research 188:20/1-17 Available from https://proceedings.mlr.press/v188/miao22a.html.

Related Material