Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning

Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, Jinwoo Shin
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5757-5766, 2020.

Abstract

Model-based reinforcement learning (RL) enjoys several benefits, such as data-efficiency and planning, by learning a model of the environment’s dynamics. However, learning a global model that can generalize across different dynamics remains a challenge. To tackle this problem, we decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-lee20g, title = {Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning}, author = {Lee, Kimin and Seo, Younggyo and Lee, Seunghyun and Lee, Honglak and Shin, Jinwoo}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5757--5766}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/lee20g/lee20g.pdf}, url = {https://proceedings.mlr.press/v119/lee20g.html}, abstract = {Model-based reinforcement learning (RL) enjoys several benefits, such as data-efficiency and planning, by learning a model of the environment’s dynamics. However, learning a global model that can generalize across different dynamics remains a challenge. To tackle this problem, we decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.} }
Endnote
%0 Conference Paper %T Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning %A Kimin Lee %A Younggyo Seo %A Seunghyun Lee %A Honglak Lee %A Jinwoo Shin %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-lee20g %I PMLR %P 5757--5766 %U https://proceedings.mlr.press/v119/lee20g.html %V 119 %X Model-based reinforcement learning (RL) enjoys several benefits, such as data-efficiency and planning, by learning a model of the environment’s dynamics. However, learning a global model that can generalize across different dynamics remains a challenge. To tackle this problem, we decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.
APA
Lee, K., Seo, Y., Lee, S., Lee, H. & Shin, J.. (2020). Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5757-5766 Available from https://proceedings.mlr.press/v119/lee20g.html.

Related Material