Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control

Yangchen Pan, Amir-massoud Farahmand, Martha White, Saleh Nabi, Piyush Grover, Daniel Nikovski
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:3986-3995, 2018.

Abstract

Recent work has shown that reinforcement learning (RL) is a promising approach to control dynamical systems described by partial differential equations (PDE). This paper shows how to use RL to tackle more general PDE control problems that have continuous high-dimensional action spaces with spatial relationship among action dimensions. In particular, we propose the concept of action descriptors, which encode regularities among spatially-extended action dimensions and enable the agent to control high-dimensional action PDEs. We provide theoretical evidence suggesting that this approach can be more sample efficient compared to a conventional approach that treats each action dimension separately and does not explicitly exploit the spatial regularity of the action space. The action descriptor approach is then used within the deep deterministic policy gradient algorithm. Experiments on two PDE control problems, with up to 256-dimensional continuous actions, show the advantage of the proposed approach over the conventional one.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-pan18a, title = {Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control}, author = {Pan, Yangchen and Farahmand, Amir-massoud and White, Martha and Nabi, Saleh and Grover, Piyush and Nikovski, Daniel}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {3986--3995}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/pan18a/pan18a.pdf}, url = {https://proceedings.mlr.press/v80/pan18a.html}, abstract = {Recent work has shown that reinforcement learning (RL) is a promising approach to control dynamical systems described by partial differential equations (PDE). This paper shows how to use RL to tackle more general PDE control problems that have continuous high-dimensional action spaces with spatial relationship among action dimensions. In particular, we propose the concept of action descriptors, which encode regularities among spatially-extended action dimensions and enable the agent to control high-dimensional action PDEs. We provide theoretical evidence suggesting that this approach can be more sample efficient compared to a conventional approach that treats each action dimension separately and does not explicitly exploit the spatial regularity of the action space. The action descriptor approach is then used within the deep deterministic policy gradient algorithm. Experiments on two PDE control problems, with up to 256-dimensional continuous actions, show the advantage of the proposed approach over the conventional one.} }
Endnote
%0 Conference Paper %T Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control %A Yangchen Pan %A Amir-massoud Farahmand %A Martha White %A Saleh Nabi %A Piyush Grover %A Daniel Nikovski %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-pan18a %I PMLR %P 3986--3995 %U https://proceedings.mlr.press/v80/pan18a.html %V 80 %X Recent work has shown that reinforcement learning (RL) is a promising approach to control dynamical systems described by partial differential equations (PDE). This paper shows how to use RL to tackle more general PDE control problems that have continuous high-dimensional action spaces with spatial relationship among action dimensions. In particular, we propose the concept of action descriptors, which encode regularities among spatially-extended action dimensions and enable the agent to control high-dimensional action PDEs. We provide theoretical evidence suggesting that this approach can be more sample efficient compared to a conventional approach that treats each action dimension separately and does not explicitly exploit the spatial regularity of the action space. The action descriptor approach is then used within the deep deterministic policy gradient algorithm. Experiments on two PDE control problems, with up to 256-dimensional continuous actions, show the advantage of the proposed approach over the conventional one.
APA
Pan, Y., Farahmand, A., White, M., Nabi, S., Grover, P. & Nikovski, D.. (2018). Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:3986-3995 Available from https://proceedings.mlr.press/v80/pan18a.html.

Related Material