Transformers Are Adaptable Task Planners

Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk, Akshara Rai
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1011-1037, 2023.

Abstract

Every home is different, and every person likes things done in their particular way. Therefore, home robots of the future need to both reason about the sequential nature of day-to-day tasks and generalize to user’s preferences. To this end, we propose a Transformer Task Planner (TTP) that learns high-level actions from demonstrations by leveraging object attribute-based representations. TTP can be pre-trained on multiple preferences and shows generalization to unseen preferences using a single demonstration as a prompt in a simulated dishwasher loading task. Further, we demonstrate real-world dish rearrangement using TTP with a Franka Panda robotic arm, prompted using a single human demonstration.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-jain23a, title = {Transformers Are Adaptable Task Planners}, author = {Jain, Vidhi and Lin, Yixin and Undersander, Eric and Bisk, Yonatan and Rai, Akshara}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1011--1037}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/jain23a/jain23a.pdf}, url = {https://proceedings.mlr.press/v205/jain23a.html}, abstract = {Every home is different, and every person likes things done in their particular way. Therefore, home robots of the future need to both reason about the sequential nature of day-to-day tasks and generalize to user’s preferences. To this end, we propose a Transformer Task Planner (TTP) that learns high-level actions from demonstrations by leveraging object attribute-based representations. TTP can be pre-trained on multiple preferences and shows generalization to unseen preferences using a single demonstration as a prompt in a simulated dishwasher loading task. Further, we demonstrate real-world dish rearrangement using TTP with a Franka Panda robotic arm, prompted using a single human demonstration.} }
Endnote
%0 Conference Paper %T Transformers Are Adaptable Task Planners %A Vidhi Jain %A Yixin Lin %A Eric Undersander %A Yonatan Bisk %A Akshara Rai %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-jain23a %I PMLR %P 1011--1037 %U https://proceedings.mlr.press/v205/jain23a.html %V 205 %X Every home is different, and every person likes things done in their particular way. Therefore, home robots of the future need to both reason about the sequential nature of day-to-day tasks and generalize to user’s preferences. To this end, we propose a Transformer Task Planner (TTP) that learns high-level actions from demonstrations by leveraging object attribute-based representations. TTP can be pre-trained on multiple preferences and shows generalization to unseen preferences using a single demonstration as a prompt in a simulated dishwasher loading task. Further, we demonstrate real-world dish rearrangement using TTP with a Franka Panda robotic arm, prompted using a single human demonstration.
APA
Jain, V., Lin, Y., Undersander, E., Bisk, Y. & Rai, A.. (2023). Transformers Are Adaptable Task Planners. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1011-1037 Available from https://proceedings.mlr.press/v205/jain23a.html.

Related Material