Goal-Aware Prediction: Learning to Model What Matters

Suraj Nair, Silvio Savarese, Chelsea Finn
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7207-7219, 2020.

Abstract

Learned dynamics models combined with both planning and policy learning algorithms have shown promise in enabling artificial agents to learn to perform many diverse tasks with limited supervision. However, one of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model (future state reconstruction), and that of the downstream planner or policy (completing a specified task). This issue is exacerbated by vision-based control tasks in diverse real-world environments, where the complexity of the real world dwarfs model capacity. In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task. Further, we do so in an entirely self-supervised manner, without the need for a reward function or image labels. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-nair20a, title = {Goal-Aware Prediction: Learning to Model What Matters}, author = {Nair, Suraj and Savarese, Silvio and Finn, Chelsea}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {7207--7219}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/nair20a/nair20a.pdf}, url = {https://proceedings.mlr.press/v119/nair20a.html}, abstract = {Learned dynamics models combined with both planning and policy learning algorithms have shown promise in enabling artificial agents to learn to perform many diverse tasks with limited supervision. However, one of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model (future state reconstruction), and that of the downstream planner or policy (completing a specified task). This issue is exacerbated by vision-based control tasks in diverse real-world environments, where the complexity of the real world dwarfs model capacity. In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task. Further, we do so in an entirely self-supervised manner, without the need for a reward function or image labels. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.} }
Endnote
%0 Conference Paper %T Goal-Aware Prediction: Learning to Model What Matters %A Suraj Nair %A Silvio Savarese %A Chelsea Finn %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-nair20a %I PMLR %P 7207--7219 %U https://proceedings.mlr.press/v119/nair20a.html %V 119 %X Learned dynamics models combined with both planning and policy learning algorithms have shown promise in enabling artificial agents to learn to perform many diverse tasks with limited supervision. However, one of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model (future state reconstruction), and that of the downstream planner or policy (completing a specified task). This issue is exacerbated by vision-based control tasks in diverse real-world environments, where the complexity of the real world dwarfs model capacity. In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task. Further, we do so in an entirely self-supervised manner, without the need for a reward function or image labels. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
APA
Nair, S., Savarese, S. & Finn, C.. (2020). Goal-Aware Prediction: Learning to Model What Matters. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:7207-7219 Available from https://proceedings.mlr.press/v119/nair20a.html.

Related Material