End-to-End Differentiable Adversarial Imitation Learning

[edit]

Nir Baram, Oron Anschel, Itai Caspi, Shie Mannor ;
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:390-399, 2017.

Abstract

Generative Adversarial Networks (GANs) have been successfully applied to the problem of policy imitation in a model-free setup. However, the computation graph of GANs, that include a stochastic policy as the generative model, is no longer differentiable end-to-end, which requires the use of high-variance gradient estimation. In this paper, we introduce the Model-based Generative Adversarial Imitation Learning (MGAIL) algorithm. We show how to use a forward model to make the computation fully differentiable, which enables training policies using the exact gradient of the discriminator. The resulting algorithm trains competent policies using relatively fewer expert samples and interactions with the environment. We test it on both discrete and continuous action domains and report results that surpass the state-of-the-art.

Related Material