Data-Efficient Policy Evaluation Through Behavior Policy Search

Josiah P. Hanna, Philip S. Thomas, Peter Stone, Scott Niekum
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1394-1403, 2017.

Abstract

We consider the task of evaluating a policy for a Markov decision process (MDP). The standard unbiased technique for evaluating a policy is to deploy the policy and observe its performance. We show that the data collected from deploying a different policy, commonly called the behavior policy, can be used to produce unbiased estimates with lower mean squared error than this standard technique. We derive an analytic expression for the optimal behavior policy — the behavior policy that minimizes the mean squared error of the resulting estimates. Because this expression depends on terms that are unknown in practice, we propose a novel policy evaluation sub-problem, behavior policy search: searching for a behavior policy that reduces mean squared error. We present a behavior policy search algorithm and empirically demonstrate its effectiveness in lowering the mean squared error of policy performance estimates.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-hanna17a, title = {Data-Efficient Policy Evaluation Through Behavior Policy Search}, author = {Josiah P. Hanna and Philip S. Thomas and Peter Stone and Scott Niekum}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {1394--1403}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/hanna17a/hanna17a.pdf}, url = {https://proceedings.mlr.press/v70/hanna17a.html}, abstract = {We consider the task of evaluating a policy for a Markov decision process (MDP). The standard unbiased technique for evaluating a policy is to deploy the policy and observe its performance. We show that the data collected from deploying a different policy, commonly called the behavior policy, can be used to produce unbiased estimates with lower mean squared error than this standard technique. We derive an analytic expression for the optimal behavior policy — the behavior policy that minimizes the mean squared error of the resulting estimates. Because this expression depends on terms that are unknown in practice, we propose a novel policy evaluation sub-problem, behavior policy search: searching for a behavior policy that reduces mean squared error. We present a behavior policy search algorithm and empirically demonstrate its effectiveness in lowering the mean squared error of policy performance estimates.} }
Endnote
%0 Conference Paper %T Data-Efficient Policy Evaluation Through Behavior Policy Search %A Josiah P. Hanna %A Philip S. Thomas %A Peter Stone %A Scott Niekum %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-hanna17a %I PMLR %P 1394--1403 %U https://proceedings.mlr.press/v70/hanna17a.html %V 70 %X We consider the task of evaluating a policy for a Markov decision process (MDP). The standard unbiased technique for evaluating a policy is to deploy the policy and observe its performance. We show that the data collected from deploying a different policy, commonly called the behavior policy, can be used to produce unbiased estimates with lower mean squared error than this standard technique. We derive an analytic expression for the optimal behavior policy — the behavior policy that minimizes the mean squared error of the resulting estimates. Because this expression depends on terms that are unknown in practice, we propose a novel policy evaluation sub-problem, behavior policy search: searching for a behavior policy that reduces mean squared error. We present a behavior policy search algorithm and empirically demonstrate its effectiveness in lowering the mean squared error of policy performance estimates.
APA
Hanna, J.P., Thomas, P.S., Stone, P. & Niekum, S.. (2017). Data-Efficient Policy Evaluation Through Behavior Policy Search. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:1394-1403 Available from https://proceedings.mlr.press/v70/hanna17a.html.

Related Material