Interactive Learning from Policy-Dependent Human Feedback

James MacGlashan, Mark K. Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, Michael L. Littman
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:2285-2294, 2017.

Abstract

This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner’s current policy. We present empirical results that show this assumption to be false—whether human trainers give a positive or negative feedback for a decision is influenced by the learner’s current policy. Based on this insight, we introduce Convergent Actor-Critic by Humans (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-macglashan17a, title = {Interactive Learning from Policy-Dependent Human Feedback}, author = {James MacGlashan and Mark K. Ho and Robert Loftin and Bei Peng and Guan Wang and David L. Roberts and Matthew E. Taylor and Michael L. Littman}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {2285--2294}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/macglashan17a/macglashan17a.pdf}, url = {https://proceedings.mlr.press/v70/macglashan17a.html}, abstract = {This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner’s current policy. We present empirical results that show this assumption to be false—whether human trainers give a positive or negative feedback for a decision is influenced by the learner’s current policy. Based on this insight, we introduce Convergent Actor-Critic by Humans (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.} }
Endnote
%0 Conference Paper %T Interactive Learning from Policy-Dependent Human Feedback %A James MacGlashan %A Mark K. Ho %A Robert Loftin %A Bei Peng %A Guan Wang %A David L. Roberts %A Matthew E. Taylor %A Michael L. Littman %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-macglashan17a %I PMLR %P 2285--2294 %U https://proceedings.mlr.press/v70/macglashan17a.html %V 70 %X This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner’s current policy. We present empirical results that show this assumption to be false—whether human trainers give a positive or negative feedback for a decision is influenced by the learner’s current policy. Based on this insight, we introduce Convergent Actor-Critic by Humans (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.
APA
MacGlashan, J., Ho, M.K., Loftin, R., Peng, B., Wang, G., Roberts, D.L., Taylor, M.E. & Littman, M.L.. (2017). Interactive Learning from Policy-Dependent Human Feedback. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:2285-2294 Available from https://proceedings.mlr.press/v70/macglashan17a.html.

Related Material