ValueAware Loss Function for Modelbased Reinforcement Learning
[edit]
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:14861494, 2017.
Abstract
We consider the problem of estimating the transition probability kernel to be used by a modelbased reinforcement learning (RL) algorithm. We argue that estimating a generative model that minimizes a probabilistic loss, such as the logloss, is an overkill because it does not take into account the underlying structure of decision problem and the RL algorithm that intends to solve it. We introduce a loss function that takes the structure of the value function into account. We provide a finitesample upper bound for the loss function showing the dependence of the error on model approximation error, number of samples, and the complexity of the model space. We also empirically compare the method with the maximum likelihood estimator on a simple problem.
Related Material


