Risk-Sensitive Generative Adversarial Imitation Learning

Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, Marco Pavone
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:2154-2163, 2019.

Abstract

We study risk-sensitive imitation learning where the agent’s goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the generative adversarial approach to imitation learning (GAIL) and derive an optimization problem for our formulation, which we call it risk- sensitive GAIL (RS-GAIL). We then derive two different versions of our RS-GAIL optimization problem that aim at matching the risk profiles of the agent and the expert w.r.t. Jensen-Shannon (JS) divergence and Wasserstein distance, and develop risk-sensitive generative adversarial imitation learning algorithms based on these optimization problems. We evaluate the performance of our algorithms and compare them with GAIL and the risk-averse imitation learning (RAIL) algorithms in two MuJoCo and two OpenAI classical control tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-lacotte19a, title = {Risk-Sensitive Generative Adversarial Imitation Learning}, author = {Lacotte, Jonathan and Ghavamzadeh, Mohammad and Chow, Yinlam and Pavone, Marco}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {2154--2163}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/lacotte19a/lacotte19a.pdf}, url = {https://proceedings.mlr.press/v89/lacotte19a.html}, abstract = {We study risk-sensitive imitation learning where the agent’s goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the generative adversarial approach to imitation learning (GAIL) and derive an optimization problem for our formulation, which we call it risk- sensitive GAIL (RS-GAIL). We then derive two different versions of our RS-GAIL optimization problem that aim at matching the risk profiles of the agent and the expert w.r.t. Jensen-Shannon (JS) divergence and Wasserstein distance, and develop risk-sensitive generative adversarial imitation learning algorithms based on these optimization problems. We evaluate the performance of our algorithms and compare them with GAIL and the risk-averse imitation learning (RAIL) algorithms in two MuJoCo and two OpenAI classical control tasks.} }
Endnote
%0 Conference Paper %T Risk-Sensitive Generative Adversarial Imitation Learning %A Jonathan Lacotte %A Mohammad Ghavamzadeh %A Yinlam Chow %A Marco Pavone %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-lacotte19a %I PMLR %P 2154--2163 %U https://proceedings.mlr.press/v89/lacotte19a.html %V 89 %X We study risk-sensitive imitation learning where the agent’s goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the generative adversarial approach to imitation learning (GAIL) and derive an optimization problem for our formulation, which we call it risk- sensitive GAIL (RS-GAIL). We then derive two different versions of our RS-GAIL optimization problem that aim at matching the risk profiles of the agent and the expert w.r.t. Jensen-Shannon (JS) divergence and Wasserstein distance, and develop risk-sensitive generative adversarial imitation learning algorithms based on these optimization problems. We evaluate the performance of our algorithms and compare them with GAIL and the risk-averse imitation learning (RAIL) algorithms in two MuJoCo and two OpenAI classical control tasks.
APA
Lacotte, J., Ghavamzadeh, M., Chow, Y. & Pavone, M.. (2019). Risk-Sensitive Generative Adversarial Imitation Learning. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:2154-2163 Available from https://proceedings.mlr.press/v89/lacotte19a.html.

Related Material