Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations

Haoran Xu, Xianyuan Zhan, Honglei Yin, Huiling Qin
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:24725-24742, 2022.

Abstract

We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with a supplementary offline dataset from suboptimal behaviors. Prior works that address this problem either require that expert data occupies the majority proportion of the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) afterwards. In this paper, we aim to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations contain a large proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator to distinguish expert and non-expert data. We propose a cooperation framework to boost the learning of both tasks, Based on this framework, we design a new IL algorithm, where the outputs of the discriminator serve as the weights of the BC loss. Experimental results show that our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-xu22l, title = {Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations}, author = {Xu, Haoran and Zhan, Xianyuan and Yin, Honglei and Qin, Huiling}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {24725--24742}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/xu22l/xu22l.pdf}, url = {https://proceedings.mlr.press/v162/xu22l.html}, abstract = {We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with a supplementary offline dataset from suboptimal behaviors. Prior works that address this problem either require that expert data occupies the majority proportion of the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) afterwards. In this paper, we aim to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations contain a large proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator to distinguish expert and non-expert data. We propose a cooperation framework to boost the learning of both tasks, Based on this framework, we design a new IL algorithm, where the outputs of the discriminator serve as the weights of the BC loss. Experimental results show that our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.} }
Endnote
%0 Conference Paper %T Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations %A Haoran Xu %A Xianyuan Zhan %A Honglei Yin %A Huiling Qin %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-xu22l %I PMLR %P 24725--24742 %U https://proceedings.mlr.press/v162/xu22l.html %V 162 %X We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with a supplementary offline dataset from suboptimal behaviors. Prior works that address this problem either require that expert data occupies the majority proportion of the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) afterwards. In this paper, we aim to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations contain a large proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator to distinguish expert and non-expert data. We propose a cooperation framework to boost the learning of both tasks, Based on this framework, we design a new IL algorithm, where the outputs of the discriminator serve as the weights of the BC loss. Experimental results show that our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.
APA
Xu, H., Zhan, X., Yin, H. & Qin, H.. (2022). Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:24725-24742 Available from https://proceedings.mlr.press/v162/xu22l.html.

Related Material