Provably Efficient Model-based Policy Adaptation

Yuda Song, Aditi Mavalankar, Wen Sun, Sicun Gao
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:9088-9098, 2020.

Abstract

The high sample complexity of reinforcement learning challenges its use in practice. A promising approach is to quickly adapt pre-trained policies to new environments. Existing methods for this policy adaptation problem typically rely on domain randomization and meta-learning, by sampling from some distribution of target environments during pre-training, and thus face difficulty on out-of-distribution target environments. We propose new model-based mechanisms that are able to make online adaptation in unseen target environments, by combining ideas from no-regret online learning and adaptive control. We prove that the approach learns policies in the target environment that can quickly recover trajectories from the source environment, and establish the rate of convergence in general settings. We demonstrate the benefits of our approach for policy adaptation in a diverse set of continuous control tasks, achieving the performance of state-of-the-art methods with much lower sample complexity. Our project website, including code, can be found at https://yudasong.github.io/PADA.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-song20b, title = {Provably Efficient Model-based Policy Adaptation}, author = {Song, Yuda and Mavalankar, Aditi and Sun, Wen and Gao, Sicun}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {9088--9098}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/song20b/song20b.pdf}, url = {https://proceedings.mlr.press/v119/song20b.html}, abstract = {The high sample complexity of reinforcement learning challenges its use in practice. A promising approach is to quickly adapt pre-trained policies to new environments. Existing methods for this policy adaptation problem typically rely on domain randomization and meta-learning, by sampling from some distribution of target environments during pre-training, and thus face difficulty on out-of-distribution target environments. We propose new model-based mechanisms that are able to make online adaptation in unseen target environments, by combining ideas from no-regret online learning and adaptive control. We prove that the approach learns policies in the target environment that can quickly recover trajectories from the source environment, and establish the rate of convergence in general settings. We demonstrate the benefits of our approach for policy adaptation in a diverse set of continuous control tasks, achieving the performance of state-of-the-art methods with much lower sample complexity. Our project website, including code, can be found at https://yudasong.github.io/PADA.} }
Endnote
%0 Conference Paper %T Provably Efficient Model-based Policy Adaptation %A Yuda Song %A Aditi Mavalankar %A Wen Sun %A Sicun Gao %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-song20b %I PMLR %P 9088--9098 %U https://proceedings.mlr.press/v119/song20b.html %V 119 %X The high sample complexity of reinforcement learning challenges its use in practice. A promising approach is to quickly adapt pre-trained policies to new environments. Existing methods for this policy adaptation problem typically rely on domain randomization and meta-learning, by sampling from some distribution of target environments during pre-training, and thus face difficulty on out-of-distribution target environments. We propose new model-based mechanisms that are able to make online adaptation in unseen target environments, by combining ideas from no-regret online learning and adaptive control. We prove that the approach learns policies in the target environment that can quickly recover trajectories from the source environment, and establish the rate of convergence in general settings. We demonstrate the benefits of our approach for policy adaptation in a diverse set of continuous control tasks, achieving the performance of state-of-the-art methods with much lower sample complexity. Our project website, including code, can be found at https://yudasong.github.io/PADA.
APA
Song, Y., Mavalankar, A., Sun, W. & Gao, S.. (2020). Provably Efficient Model-based Policy Adaptation. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:9088-9098 Available from https://proceedings.mlr.press/v119/song20b.html.

Related Material