Object-centric Forward Modeling for Model Predictive Control

Yufei Ye, Dhiraj Gandhi, Abhinav Gupta, Shubham Tulsiani
Proceedings of the Conference on Robot Learning, PMLR 100:100-109, 2020.

Abstract

We present an approach to learn an object-centric forward model, and show that this allows us to plan for sequences of actions to achieve distant desired goals. We propose to model a scene as a collection of objects, each with an explicit spatial location and implicit visual feature, and learn to model the effects of actions using random interaction data. Our model allows capturing the robot-object and object-object interactions, and leads to more sample-efficient and accurate predictions. We show that this learned model can be leveraged to search for action sequences that lead to desired goal configurations, and that in conjunction with a learned correction module, this allows for robust closed loop execution. We present experiments both in simulation and the real world, and show that our approach improves over alternate implicit or pixel-space forward models. Please see our project page for result videos.

Cite this Paper

BibTeX
@InProceedings{pmlr-v100-ye20a, title = {Object-centric Forward Modeling for Model Predictive Control}, author = {Ye, Yufei and Gandhi, Dhiraj and Gupta, Abhinav and Tulsiani, Shubham}, booktitle = {Proceedings of the Conference on Robot Learning}, pages = {100--109}, year = {2020}, editor = {Kaelbling, Leslie Pack and Kragic, Danica and Sugiura, Komei}, volume = {100}, series = {Proceedings of Machine Learning Research}, month = {30 Oct--01 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v100/ye20a/ye20a.pdf}, url = {https://proceedings.mlr.press/v100/ye20a.html}, abstract = {We present an approach to learn an object-centric forward model, and show that this allows us to plan for sequences of actions to achieve distant desired goals. We propose to model a scene as a collection of objects, each with an explicit spatial location and implicit visual feature, and learn to model the effects of actions using random interaction data. Our model allows capturing the robot-object and object-object interactions, and leads to more sample-efficient and accurate predictions. We show that this learned model can be leveraged to search for action sequences that lead to desired goal configurations, and that in conjunction with a learned correction module, this allows for robust closed loop execution. We present experiments both in simulation and the real world, and show that our approach improves over alternate implicit or pixel-space forward models. Please see our project page for result videos.} }
Endnote
%0 Conference Paper %T Object-centric Forward Modeling for Model Predictive Control %A Yufei Ye %A Dhiraj Gandhi %A Abhinav Gupta %A Shubham Tulsiani %B Proceedings of the Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2020 %E Leslie Pack Kaelbling %E Danica Kragic %E Komei Sugiura %F pmlr-v100-ye20a %I PMLR %P 100--109 %U https://proceedings.mlr.press/v100/ye20a.html %V 100 %X We present an approach to learn an object-centric forward model, and show that this allows us to plan for sequences of actions to achieve distant desired goals. We propose to model a scene as a collection of objects, each with an explicit spatial location and implicit visual feature, and learn to model the effects of actions using random interaction data. Our model allows capturing the robot-object and object-object interactions, and leads to more sample-efficient and accurate predictions. We show that this learned model can be leveraged to search for action sequences that lead to desired goal configurations, and that in conjunction with a learned correction module, this allows for robust closed loop execution. We present experiments both in simulation and the real world, and show that our approach improves over alternate implicit or pixel-space forward models. Please see our project page for result videos.
APA
Ye, Y., Gandhi, D., Gupta, A. & Tulsiani, S.. (2020). Object-centric Forward Modeling for Model Predictive Control. Proceedings of the Conference on Robot Learning, in Proceedings of Machine Learning Research 100:100-109 Available from https://proceedings.mlr.press/v100/ye20a.html.

Related Material