A New Representation of Successor Features for Transfer across Dissimilar Environments

Majid Abdolshah, Hung Le, Thommen Karimpanal George, Sunil Gupta, Santu Rana, Svetha Venkatesh
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:1-9, 2021.

Abstract

Transfer in reinforcement learning is usually achieved through generalisation across tasks. Whilst many studies have investigated transferring knowledge when the reward function changes, they have assumed that the dynamics of the environments remain consistent. Many real-world RL problems require transfer among environments with different dynamics. To address this problem, we propose an approach based on successor features in which we model successor feature functions with Gaussian Processes permitting the source successor features to be treated as noisy measurements of the target successor feature function. Our theoretical analysis proves the convergence of this approach as well as the bounded error on modelling successor feature functions with Gaussian Processes in environments with both different dynamics and rewards. We demonstrate our method on benchmark datasets and show that it outperforms current baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-abdolshah21a, title = {A New Representation of Successor Features for Transfer across Dissimilar Environments}, author = {Abdolshah, Majid and Le, Hung and George, Thommen Karimpanal and Gupta, Sunil and Rana, Santu and Venkatesh, Svetha}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {1--9}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/abdolshah21a/abdolshah21a.pdf}, url = {https://proceedings.mlr.press/v139/abdolshah21a.html}, abstract = {Transfer in reinforcement learning is usually achieved through generalisation across tasks. Whilst many studies have investigated transferring knowledge when the reward function changes, they have assumed that the dynamics of the environments remain consistent. Many real-world RL problems require transfer among environments with different dynamics. To address this problem, we propose an approach based on successor features in which we model successor feature functions with Gaussian Processes permitting the source successor features to be treated as noisy measurements of the target successor feature function. Our theoretical analysis proves the convergence of this approach as well as the bounded error on modelling successor feature functions with Gaussian Processes in environments with both different dynamics and rewards. We demonstrate our method on benchmark datasets and show that it outperforms current baselines.} }
Endnote
%0 Conference Paper %T A New Representation of Successor Features for Transfer across Dissimilar Environments %A Majid Abdolshah %A Hung Le %A Thommen Karimpanal George %A Sunil Gupta %A Santu Rana %A Svetha Venkatesh %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-abdolshah21a %I PMLR %P 1--9 %U https://proceedings.mlr.press/v139/abdolshah21a.html %V 139 %X Transfer in reinforcement learning is usually achieved through generalisation across tasks. Whilst many studies have investigated transferring knowledge when the reward function changes, they have assumed that the dynamics of the environments remain consistent. Many real-world RL problems require transfer among environments with different dynamics. To address this problem, we propose an approach based on successor features in which we model successor feature functions with Gaussian Processes permitting the source successor features to be treated as noisy measurements of the target successor feature function. Our theoretical analysis proves the convergence of this approach as well as the bounded error on modelling successor feature functions with Gaussian Processes in environments with both different dynamics and rewards. We demonstrate our method on benchmark datasets and show that it outperforms current baselines.
APA
Abdolshah, M., Le, H., George, T.K., Gupta, S., Rana, S. & Venkatesh, S.. (2021). A New Representation of Successor Features for Transfer across Dissimilar Environments. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:1-9 Available from https://proceedings.mlr.press/v139/abdolshah21a.html.

Related Material