ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting

Kushal Kedia, Prithwish Dan, Atiksh Bhardwaj, Sanjiban Choudhury
Proceedings of The 7th Conference on Robot Learning, PMLR 229:1050-1067, 2023.

Abstract

Seamless human-robot manipulation in close proximity relies on accurate forecasts of human motion. While there has been significant progress in learning forecast models at scale, when applied to manipulation tasks, these models accrue high errors at critical transition points leading to degradation in downstream planning performance. Our key insight is that instead of predicting the most likely human motion, it is sufficient to produce forecasts that capture how future human motion would affect the cost of a robot’s plan. We present ManiCast, a novel framework that learns cost-aware human forecasts and feeds them to a model predictive control planner to execute collaborative manipulation tasks. Our framework enables fluid, real-time interactions between a human and a 7-DoF robot arm across a number of real-world tasks such as reactive stirring, object handovers, and collaborative table setting. We evaluate both the motion forecasts and the end-to-end forecaster-planner system against a range of learned and heuristic baselines while additionally contributing new datasets. We release our code and datasets at https://portal-cornell.github.io/manicast/.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-kedia23a, title = {ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting}, author = {Kedia, Kushal and Dan, Prithwish and Bhardwaj, Atiksh and Choudhury, Sanjiban}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {1050--1067}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/kedia23a/kedia23a.pdf}, url = {https://proceedings.mlr.press/v229/kedia23a.html}, abstract = {Seamless human-robot manipulation in close proximity relies on accurate forecasts of human motion. While there has been significant progress in learning forecast models at scale, when applied to manipulation tasks, these models accrue high errors at critical transition points leading to degradation in downstream planning performance. Our key insight is that instead of predicting the most likely human motion, it is sufficient to produce forecasts that capture how future human motion would affect the cost of a robot’s plan. We present ManiCast, a novel framework that learns cost-aware human forecasts and feeds them to a model predictive control planner to execute collaborative manipulation tasks. Our framework enables fluid, real-time interactions between a human and a 7-DoF robot arm across a number of real-world tasks such as reactive stirring, object handovers, and collaborative table setting. We evaluate both the motion forecasts and the end-to-end forecaster-planner system against a range of learned and heuristic baselines while additionally contributing new datasets. We release our code and datasets at https://portal-cornell.github.io/manicast/.} }
Endnote
%0 Conference Paper %T ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting %A Kushal Kedia %A Prithwish Dan %A Atiksh Bhardwaj %A Sanjiban Choudhury %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-kedia23a %I PMLR %P 1050--1067 %U https://proceedings.mlr.press/v229/kedia23a.html %V 229 %X Seamless human-robot manipulation in close proximity relies on accurate forecasts of human motion. While there has been significant progress in learning forecast models at scale, when applied to manipulation tasks, these models accrue high errors at critical transition points leading to degradation in downstream planning performance. Our key insight is that instead of predicting the most likely human motion, it is sufficient to produce forecasts that capture how future human motion would affect the cost of a robot’s plan. We present ManiCast, a novel framework that learns cost-aware human forecasts and feeds them to a model predictive control planner to execute collaborative manipulation tasks. Our framework enables fluid, real-time interactions between a human and a 7-DoF robot arm across a number of real-world tasks such as reactive stirring, object handovers, and collaborative table setting. We evaluate both the motion forecasts and the end-to-end forecaster-planner system against a range of learned and heuristic baselines while additionally contributing new datasets. We release our code and datasets at https://portal-cornell.github.io/manicast/.
APA
Kedia, K., Dan, P., Bhardwaj, A. & Choudhury, S.. (2023). ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:1050-1067 Available from https://proceedings.mlr.press/v229/kedia23a.html.

Related Material