Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning

Emilio Jorge, Hannes Eriksson, Christos Dimitrakakis, Debabrota Basu, Divya Grover
Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, PMLR 137:43-52, 2020.

Abstract

Bayesian Reinforcement Learning (BRL) offers a decision-theoretic solution to the reinforcement learning problem. While “model-based” BRL algorithms have focused either on maintaining a posterior distribution on models, BRL “model-free” methods try to estimate value function distributions but make strong implicit assumptions or approximations. We describe a novel Bayesian framework, \emph{inferential induction}, for correctly inferring value function distributions from data, which leads to a new family of BRL algorithms. We design an algorithm, Bayesian Backwards Induction (BBI), with this framework. We experimentally demonstrate that BBI is competitive with the state of the art. However, its advantage relative to existing BRL model-free methods is not as great as we have expected, particularly when the additional computational burden is taken into account.

Cite this Paper

BibTeX
@InProceedings{pmlr-v137-jorge20a, title = {Inferential Induction: A Novel Framework for {B}ayesian Reinforcement Learning}, author = {Jorge, Emilio and Eriksson, Hannes and Dimitrakakis, Christos and Basu, Debabrota and Grover, Divya}, booktitle = {Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops}, pages = {43--52}, year = {2020}, editor = {Zosa Forde, Jessica and Ruiz, Francisco and Pradier, Melanie F. and Schein, Aaron}, volume = {137}, series = {Proceedings of Machine Learning Research}, month = {12 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v137/jorge20a/jorge20a.pdf}, url = {https://proceedings.mlr.press/v137/jorge20a.html}, abstract = {Bayesian Reinforcement Learning (BRL) offers a decision-theoretic solution to the reinforcement learning problem. While “model-based” BRL algorithms have focused either on maintaining a posterior distribution on models, BRL “model-free” methods try to estimate value function distributions but make strong implicit assumptions or approximations. We describe a novel Bayesian framework, \emph{inferential induction}, for correctly inferring value function distributions from data, which leads to a new family of BRL algorithms. We design an algorithm, Bayesian Backwards Induction (BBI), with this framework. We experimentally demonstrate that BBI is competitive with the state of the art. However, its advantage relative to existing BRL model-free methods is not as great as we have expected, particularly when the additional computational burden is taken into account.} }
Endnote
%0 Conference Paper %T Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning %A Emilio Jorge %A Hannes Eriksson %A Christos Dimitrakakis %A Debabrota Basu %A Divya Grover %B Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops %C Proceedings of Machine Learning Research %D 2020 %E Jessica Zosa Forde %E Francisco Ruiz %E Melanie F. Pradier %E Aaron Schein %F pmlr-v137-jorge20a %I PMLR %P 43--52 %U https://proceedings.mlr.press/v137/jorge20a.html %V 137 %X Bayesian Reinforcement Learning (BRL) offers a decision-theoretic solution to the reinforcement learning problem. While “model-based” BRL algorithms have focused either on maintaining a posterior distribution on models, BRL “model-free” methods try to estimate value function distributions but make strong implicit assumptions or approximations. We describe a novel Bayesian framework, \emph{inferential induction}, for correctly inferring value function distributions from data, which leads to a new family of BRL algorithms. We design an algorithm, Bayesian Backwards Induction (BBI), with this framework. We experimentally demonstrate that BBI is competitive with the state of the art. However, its advantage relative to existing BRL model-free methods is not as great as we have expected, particularly when the additional computational burden is taken into account.
APA
Jorge, E., Eriksson, H., Dimitrakakis, C., Basu, D. & Grover, D.. (2020). Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning. Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, in Proceedings of Machine Learning Research 137:43-52 Available from https://proceedings.mlr.press/v137/jorge20a.html.

Related Material