Curriculum Reinforcement Learning via Constrained Optimal Transport

Pascal Klink, Haoyi Yang, Carlo D’Eramo, Jan Peters, Joni Pajarinen
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:11341-11358, 2022.

Abstract

Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-klink22a, title = {Curriculum Reinforcement Learning via Constrained Optimal Transport}, author = {Klink, Pascal and Yang, Haoyi and D'Eramo, Carlo and Peters, Jan and Pajarinen, Joni}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {11341--11358}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/klink22a/klink22a.pdf}, url = {https://proceedings.mlr.press/v162/klink22a.html}, abstract = {Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.} }
Endnote
%0 Conference Paper %T Curriculum Reinforcement Learning via Constrained Optimal Transport %A Pascal Klink %A Haoyi Yang %A Carlo D’Eramo %A Jan Peters %A Joni Pajarinen %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-klink22a %I PMLR %P 11341--11358 %U https://proceedings.mlr.press/v162/klink22a.html %V 162 %X Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.
APA
Klink, P., Yang, H., D’Eramo, C., Peters, J. & Pajarinen, J.. (2022). Curriculum Reinforcement Learning via Constrained Optimal Transport. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:11341-11358 Available from https://proceedings.mlr.press/v162/klink22a.html.

Related Material