REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction

Zeyi Liu, Arpit Bahety, Shuran Song
Proceedings of The 7th Conference on Robot Learning, PMLR 229:3468-3484, 2023.

Abstract

The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, we introduce REFLECT, a framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a language-based planner to correct the failure and complete the task. To systematically evaluate the framework, we create the RoboFail dataset with a variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-liu23g, title = {REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction}, author = {Liu, Zeyi and Bahety, Arpit and Song, Shuran}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {3468--3484}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/liu23g/liu23g.pdf}, url = {https://proceedings.mlr.press/v229/liu23g.html}, abstract = {The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, we introduce REFLECT, a framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a language-based planner to correct the failure and complete the task. To systematically evaluate the framework, we create the RoboFail dataset with a variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.} }
Endnote
%0 Conference Paper %T REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction %A Zeyi Liu %A Arpit Bahety %A Shuran Song %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-liu23g %I PMLR %P 3468--3484 %U https://proceedings.mlr.press/v229/liu23g.html %V 229 %X The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, we introduce REFLECT, a framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a language-based planner to correct the failure and complete the task. To systematically evaluate the framework, we create the RoboFail dataset with a variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.
APA
Liu, Z., Bahety, A. & Song, S.. (2023). REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:3468-3484 Available from https://proceedings.mlr.press/v229/liu23g.html.

Related Material