Collect & Infer - a fresh look at data-efficient Reinforcement Learning

Martin Riedmiller, Jost Tobias Springenberg, Roland Hafner, Nicolas Heess
Proceedings of the 5th Conference on Robot Learning, PMLR 164:1736-1744, 2022.

Abstract

This position paper proposes a fresh look at Reinforcement Learning (RL) from the perspective of data-efficiency. RL has gone through three major stages: pure on-line RL where every data-point is considered only once, RL with a replay buffer where additional learning is done on a portion of the experience, and finally transition memory based RL, where, conceptually, all transitions are stored, and flexibly re-used in every update step. While inferring knowledge from all stored experience has led to a tremendous gain in data-efficiency, the question of how this data is collected has been vastly understudied. We argue that data-efficiency can only be achieved through careful consideration of both aspects. We propose to make this insight explicit via a paradigm that we call ’Collect and Infer’, which explicitly models RL as two separate but interconnected processes, concerned with data collection and knowledge inference respectively.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-riedmiller22a, title = {Collect & Infer - a fresh look at data-efficient Reinforcement Learning}, author = {Riedmiller, Martin and Springenberg, Jost Tobias and Hafner, Roland and Heess, Nicolas}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {1736--1744}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/riedmiller22a/riedmiller22a.pdf}, url = {https://proceedings.mlr.press/v164/riedmiller22a.html}, abstract = {This position paper proposes a fresh look at Reinforcement Learning (RL) from the perspective of data-efficiency. RL has gone through three major stages: pure on-line RL where every data-point is considered only once, RL with a replay buffer where additional learning is done on a portion of the experience, and finally transition memory based RL, where, conceptually, all transitions are stored, and flexibly re-used in every update step. While inferring knowledge from all stored experience has led to a tremendous gain in data-efficiency, the question of how this data is collected has been vastly understudied. We argue that data-efficiency can only be achieved through careful consideration of both aspects. We propose to make this insight explicit via a paradigm that we call ’Collect and Infer’, which explicitly models RL as two separate but interconnected processes, concerned with data collection and knowledge inference respectively. } }
Endnote
%0 Conference Paper %T Collect & Infer - a fresh look at data-efficient Reinforcement Learning %A Martin Riedmiller %A Jost Tobias Springenberg %A Roland Hafner %A Nicolas Heess %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-riedmiller22a %I PMLR %P 1736--1744 %U https://proceedings.mlr.press/v164/riedmiller22a.html %V 164 %X This position paper proposes a fresh look at Reinforcement Learning (RL) from the perspective of data-efficiency. RL has gone through three major stages: pure on-line RL where every data-point is considered only once, RL with a replay buffer where additional learning is done on a portion of the experience, and finally transition memory based RL, where, conceptually, all transitions are stored, and flexibly re-used in every update step. While inferring knowledge from all stored experience has led to a tremendous gain in data-efficiency, the question of how this data is collected has been vastly understudied. We argue that data-efficiency can only be achieved through careful consideration of both aspects. We propose to make this insight explicit via a paradigm that we call ’Collect and Infer’, which explicitly models RL as two separate but interconnected processes, concerned with data collection and knowledge inference respectively.
APA
Riedmiller, M., Springenberg, J.T., Hafner, R. & Heess, N.. (2022). Collect & Infer - a fresh look at data-efficient Reinforcement Learning. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:1736-1744 Available from https://proceedings.mlr.press/v164/riedmiller22a.html.

Related Material