Analysis of Stochastic Processes through Replay Buffers

Shirli Di-Castro, Shie Mannor, Dotan Di Castro
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:5039-5060, 2022.

Abstract

Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-di-castro22a, title = {Analysis of Stochastic Processes through Replay Buffers}, author = {Di-Castro, Shirli and Mannor, Shie and Castro, Dotan Di}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {5039--5060}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/di-castro22a/di-castro22a.pdf}, url = {https://proceedings.mlr.press/v162/di-castro22a.html}, abstract = {Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.} }
Endnote
%0 Conference Paper %T Analysis of Stochastic Processes through Replay Buffers %A Shirli Di-Castro %A Shie Mannor %A Dotan Di Castro %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-di-castro22a %I PMLR %P 5039--5060 %U https://proceedings.mlr.press/v162/di-castro22a.html %V 162 %X Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.
APA
Di-Castro, S., Mannor, S. & Castro, D.D.. (2022). Analysis of Stochastic Processes through Replay Buffers. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:5039-5060 Available from https://proceedings.mlr.press/v162/di-castro22a.html.

Related Material