Imitation by Predicting Observations

Andrew Jaegle, Yury Sulsky, Arun Ahuja, Jake Bruce, Rob Fergus, Greg Wayne
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:4665-4676, 2021.

Abstract

Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior. Although it is easy to observe behavior in the real-world, the underlying actions may not be accessible. We present a new method for imitation solely from observations that achieves comparable performance to experts on challenging continuous control tasks while also exhibiting robustness in the presence of observations unrelated to the task. Our method, which we call FORM (for "Future Observation Reward Model") is derived from an inverse RL objective and imitates using a model of expert behavior learned by generative modelling of the expert’s observations, without needing ground truth actions. We show that FORM performs comparably to a strong baseline IRL method (GAIL) on the DeepMind Control Suite benchmark, while outperforming GAIL in the presence of task-irrelevant features.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-jaegle21b, title = {Imitation by Predicting Observations}, author = {Jaegle, Andrew and Sulsky, Yury and Ahuja, Arun and Bruce, Jake and Fergus, Rob and Wayne, Greg}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {4665--4676}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/jaegle21b/jaegle21b.pdf}, url = {https://proceedings.mlr.press/v139/jaegle21b.html}, abstract = {Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior. Although it is easy to observe behavior in the real-world, the underlying actions may not be accessible. We present a new method for imitation solely from observations that achieves comparable performance to experts on challenging continuous control tasks while also exhibiting robustness in the presence of observations unrelated to the task. Our method, which we call FORM (for "Future Observation Reward Model") is derived from an inverse RL objective and imitates using a model of expert behavior learned by generative modelling of the expert’s observations, without needing ground truth actions. We show that FORM performs comparably to a strong baseline IRL method (GAIL) on the DeepMind Control Suite benchmark, while outperforming GAIL in the presence of task-irrelevant features.} }
Endnote
%0 Conference Paper %T Imitation by Predicting Observations %A Andrew Jaegle %A Yury Sulsky %A Arun Ahuja %A Jake Bruce %A Rob Fergus %A Greg Wayne %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-jaegle21b %I PMLR %P 4665--4676 %U https://proceedings.mlr.press/v139/jaegle21b.html %V 139 %X Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior. Although it is easy to observe behavior in the real-world, the underlying actions may not be accessible. We present a new method for imitation solely from observations that achieves comparable performance to experts on challenging continuous control tasks while also exhibiting robustness in the presence of observations unrelated to the task. Our method, which we call FORM (for "Future Observation Reward Model") is derived from an inverse RL objective and imitates using a model of expert behavior learned by generative modelling of the expert’s observations, without needing ground truth actions. We show that FORM performs comparably to a strong baseline IRL method (GAIL) on the DeepMind Control Suite benchmark, while outperforming GAIL in the presence of task-irrelevant features.
APA
Jaegle, A., Sulsky, Y., Ahuja, A., Bruce, J., Fergus, R. & Wayne, G.. (2021). Imitation by Predicting Observations. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:4665-4676 Available from https://proceedings.mlr.press/v139/jaegle21b.html.

Related Material