How to Leverage Diverse Demonstrations in Offline Imitation Learning

Sheng Yue, Jiani Liu, Xingyuan Hua, Ju Ren, Sen Lin, Junshan Zhang, Yaoxue Zhang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:58037-58067, 2024.

Abstract

Offline Imitation Learning (IL) with imperfect demonstrations has garnered increasing attention owing to the scarcity of expert data in many real-world domains. A fundamental problem in this scenario is how to extract positive behaviors from noisy data. In general, current approaches to the problem select data building on state-action similarity to given expert demonstrations, neglecting precious information in (potentially abundant) diverse state-actions that deviate from expert ones. In this paper, we introduce a simple yet effective data selection method that identifies positive behaviors based on their resultant states - a more informative criterion enabling explicit utilization of dynamics information and effective extraction of both expert and beneficial diverse behaviors. Further, we devise a lightweight behavior cloning algorithm capable of leveraging the expert and selected data correctly. In the experiments, we evaluate our method on a suite of complex and high-dimensional offline IL benchmarks, including continuous-control and vision-based tasks. The results demonstrate that our method achieves state-of-the-art performance, outperforming existing methods on 20/21 benchmarks, typically by 2-5x, while maintaining a comparable runtime to Behavior Cloning (BC).

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-yue24c, title = {How to Leverage Diverse Demonstrations in Offline Imitation Learning}, author = {Yue, Sheng and Liu, Jiani and Hua, Xingyuan and Ren, Ju and Lin, Sen and Zhang, Junshan and Zhang, Yaoxue}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {58037--58067}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/yue24c/yue24c.pdf}, url = {https://proceedings.mlr.press/v235/yue24c.html}, abstract = {Offline Imitation Learning (IL) with imperfect demonstrations has garnered increasing attention owing to the scarcity of expert data in many real-world domains. A fundamental problem in this scenario is how to extract positive behaviors from noisy data. In general, current approaches to the problem select data building on state-action similarity to given expert demonstrations, neglecting precious information in (potentially abundant) diverse state-actions that deviate from expert ones. In this paper, we introduce a simple yet effective data selection method that identifies positive behaviors based on their resultant states - a more informative criterion enabling explicit utilization of dynamics information and effective extraction of both expert and beneficial diverse behaviors. Further, we devise a lightweight behavior cloning algorithm capable of leveraging the expert and selected data correctly. In the experiments, we evaluate our method on a suite of complex and high-dimensional offline IL benchmarks, including continuous-control and vision-based tasks. The results demonstrate that our method achieves state-of-the-art performance, outperforming existing methods on 20/21 benchmarks, typically by 2-5x, while maintaining a comparable runtime to Behavior Cloning (BC).} }
Endnote
%0 Conference Paper %T How to Leverage Diverse Demonstrations in Offline Imitation Learning %A Sheng Yue %A Jiani Liu %A Xingyuan Hua %A Ju Ren %A Sen Lin %A Junshan Zhang %A Yaoxue Zhang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-yue24c %I PMLR %P 58037--58067 %U https://proceedings.mlr.press/v235/yue24c.html %V 235 %X Offline Imitation Learning (IL) with imperfect demonstrations has garnered increasing attention owing to the scarcity of expert data in many real-world domains. A fundamental problem in this scenario is how to extract positive behaviors from noisy data. In general, current approaches to the problem select data building on state-action similarity to given expert demonstrations, neglecting precious information in (potentially abundant) diverse state-actions that deviate from expert ones. In this paper, we introduce a simple yet effective data selection method that identifies positive behaviors based on their resultant states - a more informative criterion enabling explicit utilization of dynamics information and effective extraction of both expert and beneficial diverse behaviors. Further, we devise a lightweight behavior cloning algorithm capable of leveraging the expert and selected data correctly. In the experiments, we evaluate our method on a suite of complex and high-dimensional offline IL benchmarks, including continuous-control and vision-based tasks. The results demonstrate that our method achieves state-of-the-art performance, outperforming existing methods on 20/21 benchmarks, typically by 2-5x, while maintaining a comparable runtime to Behavior Cloning (BC).
APA
Yue, S., Liu, J., Hua, X., Ren, J., Lin, S., Zhang, J. & Zhang, Y.. (2024). How to Leverage Diverse Demonstrations in Offline Imitation Learning. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:58037-58067 Available from https://proceedings.mlr.press/v235/yue24c.html.

Related Material