Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation

Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, Yiannis Demiris
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6536-6544, 2019.

Abstract

We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert’s reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, which allows us to re-frame imitation learning within the standard reinforcement learning setting. We demonstrate the efficacy of our reward function on both discrete and continuous domains, achieving comparable or better performance than the state of the art under different reinforcement learning algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-wang19d, title = {Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation}, author = {Wang, Ruohan and Ciliberto, Carlo and Amadori, Pierluigi Vito and Demiris, Yiannis}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {6536--6544}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/wang19d/wang19d.pdf}, url = {https://proceedings.mlr.press/v97/wang19d.html}, abstract = {We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert’s reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, which allows us to re-frame imitation learning within the standard reinforcement learning setting. We demonstrate the efficacy of our reward function on both discrete and continuous domains, achieving comparable or better performance than the state of the art under different reinforcement learning algorithms.} }
Endnote
%0 Conference Paper %T Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation %A Ruohan Wang %A Carlo Ciliberto %A Pierluigi Vito Amadori %A Yiannis Demiris %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-wang19d %I PMLR %P 6536--6544 %U https://proceedings.mlr.press/v97/wang19d.html %V 97 %X We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert’s reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, which allows us to re-frame imitation learning within the standard reinforcement learning setting. We demonstrate the efficacy of our reward function on both discrete and continuous domains, achieving comparable or better performance than the state of the art under different reinforcement learning algorithms.
APA
Wang, R., Ciliberto, C., Amadori, P.V. & Demiris, Y.. (2019). Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:6536-6544 Available from https://proceedings.mlr.press/v97/wang19d.html.

Related Material