Imitation Learning from Imperfect Demonstration

Yueh-Hua Wu; Nontawat Charoenphakdee; Han Bao; Voot Tangkaratt; Masashi Sugiyama

Imitation Learning from Imperfect Demonstration

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, Masashi Sugiyama ;

Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6818-6827, 2019.

Abstract

Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, we propose a novel approach that utilizes confidence scores, which describe the quality of demonstrations. More specifically, we propose two confidence-based IL methods, namely two-step importance weighting IL (2IWIL) and generative adversarial IL with imperfect demonstration and confidence (IC-GAIL). We show that confidence scores given only to a small portion of sub-optimal demonstrations significantly improve the performance of IL both theoretically and empirically.

Related Material

@InProceedings{pmlr-v97-wu19a, title = {Imitation Learning from Imperfect Demonstration}, author = {Wu, Yueh-Hua and Charoenphakdee, Nontawat and Bao, Han and Tangkaratt, Voot and Sugiyama, Masashi}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {6818--6827}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, address = {Long Beach, California, USA}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/wu19a/wu19a.pdf}, url = {http://proceedings.mlr.press/v97/wu19a.html}, abstract = {Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, we propose a novel approach that utilizes confidence scores, which describe the quality of demonstrations. More specifically, we propose two confidence-based IL methods, namely two-step importance weighting IL (2IWIL) and generative adversarial IL with imperfect demonstration and confidence (IC-GAIL). We show that confidence scores given only to a small portion of sub-optimal demonstrations significantly improve the performance of IL both theoretically and empirically.} }

%0 Conference Paper %T Imitation Learning from Imperfect Demonstration %A Yueh-Hua Wu %A Nontawat Charoenphakdee %A Han Bao %A Voot Tangkaratt %A Masashi Sugiyama %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-wu19a %I PMLR %J Proceedings of Machine Learning Research %P 6818--6827 %U http://proceedings.mlr.press %V 97 %W PMLR %X Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, we propose a novel approach that utilizes confidence scores, which describe the quality of demonstrations. More specifically, we propose two confidence-based IL methods, namely two-step importance weighting IL (2IWIL) and generative adversarial IL with imperfect demonstration and confidence (IC-GAIL). We show that confidence scores given only to a small portion of sub-optimal demonstrations significantly improve the performance of IL both theoretically and empirically.

Wu, Y., Charoenphakdee, N., Bao, H., Tangkaratt, V. & Sugiyama, M.. (2019). Imitation Learning from Imperfect Demonstration. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:6818-6827