Intrinsic Reward Driven Imitation Learning via Generative Model

Xingrui Yu, Yueming Lyu, Ivor Tsang
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:10925-10935, 2020.

Abstract

Imitation learning in a high-dimensional environment is challenging. Most inverse reinforcement learning (IRL) methods fail to outperform the demonstrator in such a high-dimensional environment, e.g., Atari domain. To address this challenge, we propose a novel reward learning module to generate intrinsic reward signals via a generative model. Our generative method can perform better forward state transition and backward action encoding, which improves the module’s dynamics modeling ability in the environment. Thus, our module provides the imitation agent both the intrinsic intention of the demonstrator and a better exploration ability, which is critical for the agent to outperform the demonstrator. Empirical results show that our method outperforms state-of-the-art IRL methods on multiple Atari games, even with one-life demonstration. Remarkably, our method achieves performance that is up to 5 times the performance of the demonstration.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-yu20d, title = {Intrinsic Reward Driven Imitation Learning via Generative Model}, author = {Yu, Xingrui and Lyu, Yueming and Tsang, Ivor}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {10925--10935}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/yu20d/yu20d.pdf}, url = {https://proceedings.mlr.press/v119/yu20d.html}, abstract = {Imitation learning in a high-dimensional environment is challenging. Most inverse reinforcement learning (IRL) methods fail to outperform the demonstrator in such a high-dimensional environment, e.g., Atari domain. To address this challenge, we propose a novel reward learning module to generate intrinsic reward signals via a generative model. Our generative method can perform better forward state transition and backward action encoding, which improves the module’s dynamics modeling ability in the environment. Thus, our module provides the imitation agent both the intrinsic intention of the demonstrator and a better exploration ability, which is critical for the agent to outperform the demonstrator. Empirical results show that our method outperforms state-of-the-art IRL methods on multiple Atari games, even with one-life demonstration. Remarkably, our method achieves performance that is up to 5 times the performance of the demonstration.} }
Endnote
%0 Conference Paper %T Intrinsic Reward Driven Imitation Learning via Generative Model %A Xingrui Yu %A Yueming Lyu %A Ivor Tsang %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-yu20d %I PMLR %P 10925--10935 %U https://proceedings.mlr.press/v119/yu20d.html %V 119 %X Imitation learning in a high-dimensional environment is challenging. Most inverse reinforcement learning (IRL) methods fail to outperform the demonstrator in such a high-dimensional environment, e.g., Atari domain. To address this challenge, we propose a novel reward learning module to generate intrinsic reward signals via a generative model. Our generative method can perform better forward state transition and backward action encoding, which improves the module’s dynamics modeling ability in the environment. Thus, our module provides the imitation agent both the intrinsic intention of the demonstrator and a better exploration ability, which is critical for the agent to outperform the demonstrator. Empirical results show that our method outperforms state-of-the-art IRL methods on multiple Atari games, even with one-life demonstration. Remarkably, our method achieves performance that is up to 5 times the performance of the demonstration.
APA
Yu, X., Lyu, Y. & Tsang, I.. (2020). Intrinsic Reward Driven Imitation Learning via Generative Model. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10925-10935 Available from https://proceedings.mlr.press/v119/yu20d.html.

Related Material