Federated Reinforcement Learning with Environment Heterogeneity

Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, Zhihua Zhang
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:18-37, 2022.

Abstract

We study Federated Reinforcement Learning (FedRL) problem in which $n$ agents collaboratively learn a single policy without sharing the trajectories they collected during agent-environment interaction. In this paper, we stress the constraint of environment heterogeneity, which means $n$ environments corresponding to these $n$ agents have different state-transitions. To obtain a value function or a policy function which optimizes the overall performance in all environments, we propose two algorithms, we propose two federated RL algorithms, QAvg and PAvg. We theoretically prove that these algorithms converge to suboptimal solutions, while such suboptimality depends on how heterogeneous these $n$ environments are. Moreover, we propose a heuristic that achieves personalization by embedding the $n$ environments into $n$ vectors. The personalization heuristic not only improves the training but also allows for better generalization to new environments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-jin22a, title = { Federated Reinforcement Learning with Environment Heterogeneity }, author = {Jin, Hao and Peng, Yang and Yang, Wenhao and Wang, Shusen and Zhang, Zhihua}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {18--37}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/jin22a/jin22a.pdf}, url = {https://proceedings.mlr.press/v151/jin22a.html}, abstract = { We study Federated Reinforcement Learning (FedRL) problem in which $n$ agents collaboratively learn a single policy without sharing the trajectories they collected during agent-environment interaction. In this paper, we stress the constraint of environment heterogeneity, which means $n$ environments corresponding to these $n$ agents have different state-transitions. To obtain a value function or a policy function which optimizes the overall performance in all environments, we propose two algorithms, we propose two federated RL algorithms, QAvg and PAvg. We theoretically prove that these algorithms converge to suboptimal solutions, while such suboptimality depends on how heterogeneous these $n$ environments are. Moreover, we propose a heuristic that achieves personalization by embedding the $n$ environments into $n$ vectors. The personalization heuristic not only improves the training but also allows for better generalization to new environments. } }
Endnote
%0 Conference Paper %T Federated Reinforcement Learning with Environment Heterogeneity %A Hao Jin %A Yang Peng %A Wenhao Yang %A Shusen Wang %A Zhihua Zhang %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-jin22a %I PMLR %P 18--37 %U https://proceedings.mlr.press/v151/jin22a.html %V 151 %X We study Federated Reinforcement Learning (FedRL) problem in which $n$ agents collaboratively learn a single policy without sharing the trajectories they collected during agent-environment interaction. In this paper, we stress the constraint of environment heterogeneity, which means $n$ environments corresponding to these $n$ agents have different state-transitions. To obtain a value function or a policy function which optimizes the overall performance in all environments, we propose two algorithms, we propose two federated RL algorithms, QAvg and PAvg. We theoretically prove that these algorithms converge to suboptimal solutions, while such suboptimality depends on how heterogeneous these $n$ environments are. Moreover, we propose a heuristic that achieves personalization by embedding the $n$ environments into $n$ vectors. The personalization heuristic not only improves the training but also allows for better generalization to new environments.
APA
Jin, H., Peng, Y., Yang, W., Wang, S. & Zhang, Z.. (2022). Federated Reinforcement Learning with Environment Heterogeneity . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:18-37 Available from https://proceedings.mlr.press/v151/jin22a.html.

Related Material