Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems

Miguel Suau, Jinke He, Matthijs T. J. Spaan, Frans Oliehoek
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:20604-20624, 2022.

Abstract

Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-suau22a, title = {Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep {RL} in Large Networked Systems}, author = {Suau, Miguel and He, Jinke and Spaan, Matthijs T. J. and Oliehoek, Frans}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {20604--20624}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/suau22a/suau22a.pdf}, url = {https://proceedings.mlr.press/v162/suau22a.html}, abstract = {Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.} }
Endnote
%0 Conference Paper %T Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems %A Miguel Suau %A Jinke He %A Matthijs T. J. Spaan %A Frans Oliehoek %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-suau22a %I PMLR %P 20604--20624 %U https://proceedings.mlr.press/v162/suau22a.html %V 162 %X Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.
APA
Suau, M., He, J., Spaan, M.T.J. & Oliehoek, F.. (2022). Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:20604-20624 Available from https://proceedings.mlr.press/v162/suau22a.html.

Related Material