IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors

Gaurav Datta, Ryan Hoque, Anrui Gu, Eugen Solowjow, Ken Goldberg
Proceedings of The 7th Conference on Robot Learning, PMLR 229:2340-2356, 2023.

Abstract

Imitation learning has been applied to a range of robotic tasks, but can struggle when robots encounter edge cases that are not represented in the training data (i.e., distribution shift). Interactive fleet learning (IFL) mitigates distribution shift by allowing robots to access remote human supervisors during task execution and learn from them over time, but different supervisors may demonstrate the task in different ways. Recent work proposes Implicit Behavior Cloning (IBC), which is able to represent multimodal demonstrations using energy-based models (EBMs). In this work, we propose Implicit Interactive Fleet Learning (IIFL), an algorithm that builds on IBC for interactive imitation learning from multiple heterogeneous human supervisors. A key insight in IIFL is a novel approach for uncertainty quantification in EBMs using Jeffreys divergence. While IIFL is more computationally expensive than explicit methods, results suggest that IIFL achieves a 2.8x higher success rate in simulation experiments and a 4.5x higher return on human effort in a physical block pushing task over (Explicit) IFL, IBC, and other baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-datta23a, title = {IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors}, author = {Datta, Gaurav and Hoque, Ryan and Gu, Anrui and Solowjow, Eugen and Goldberg, Ken}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {2340--2356}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/datta23a/datta23a.pdf}, url = {https://proceedings.mlr.press/v229/datta23a.html}, abstract = {Imitation learning has been applied to a range of robotic tasks, but can struggle when robots encounter edge cases that are not represented in the training data (i.e., distribution shift). Interactive fleet learning (IFL) mitigates distribution shift by allowing robots to access remote human supervisors during task execution and learn from them over time, but different supervisors may demonstrate the task in different ways. Recent work proposes Implicit Behavior Cloning (IBC), which is able to represent multimodal demonstrations using energy-based models (EBMs). In this work, we propose Implicit Interactive Fleet Learning (IIFL), an algorithm that builds on IBC for interactive imitation learning from multiple heterogeneous human supervisors. A key insight in IIFL is a novel approach for uncertainty quantification in EBMs using Jeffreys divergence. While IIFL is more computationally expensive than explicit methods, results suggest that IIFL achieves a 2.8x higher success rate in simulation experiments and a 4.5x higher return on human effort in a physical block pushing task over (Explicit) IFL, IBC, and other baselines.} }
Endnote
%0 Conference Paper %T IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors %A Gaurav Datta %A Ryan Hoque %A Anrui Gu %A Eugen Solowjow %A Ken Goldberg %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-datta23a %I PMLR %P 2340--2356 %U https://proceedings.mlr.press/v229/datta23a.html %V 229 %X Imitation learning has been applied to a range of robotic tasks, but can struggle when robots encounter edge cases that are not represented in the training data (i.e., distribution shift). Interactive fleet learning (IFL) mitigates distribution shift by allowing robots to access remote human supervisors during task execution and learn from them over time, but different supervisors may demonstrate the task in different ways. Recent work proposes Implicit Behavior Cloning (IBC), which is able to represent multimodal demonstrations using energy-based models (EBMs). In this work, we propose Implicit Interactive Fleet Learning (IIFL), an algorithm that builds on IBC for interactive imitation learning from multiple heterogeneous human supervisors. A key insight in IIFL is a novel approach for uncertainty quantification in EBMs using Jeffreys divergence. While IIFL is more computationally expensive than explicit methods, results suggest that IIFL achieves a 2.8x higher success rate in simulation experiments and a 4.5x higher return on human effort in a physical block pushing task over (Explicit) IFL, IBC, and other baselines.
APA
Datta, G., Hoque, R., Gu, A., Solowjow, E. & Goldberg, K.. (2023). IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:2340-2356 Available from https://proceedings.mlr.press/v229/datta23a.html.

Related Material