Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret

Haitham Bou Ammar, Rasul Tutunov, Eric Eaton
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:2361-2369, 2015.

Abstract

Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases, and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial setting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-ammar15, title = {Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret}, author = {Ammar, Haitham Bou and Tutunov, Rasul and Eaton, Eric}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {2361--2369}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/ammar15.pdf}, url = {https://proceedings.mlr.press/v37/ammar15.html}, abstract = {Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases, and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial setting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control.} }
Endnote
%0 Conference Paper %T Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret %A Haitham Bou Ammar %A Rasul Tutunov %A Eric Eaton %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-ammar15 %I PMLR %P 2361--2369 %U https://proceedings.mlr.press/v37/ammar15.html %V 37 %X Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases, and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial setting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control.
RIS
TY - CPAPER TI - Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret AU - Haitham Bou Ammar AU - Rasul Tutunov AU - Eric Eaton BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-ammar15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 2361 EP - 2369 L1 - http://proceedings.mlr.press/v37/ammar15.pdf UR - https://proceedings.mlr.press/v37/ammar15.html AB - Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases, and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial setting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control. ER -
APA
Ammar, H.B., Tutunov, R. & Eaton, E.. (2015). Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:2361-2369 Available from https://proceedings.mlr.press/v37/ammar15.html.

Related Material