Interactive Learning from Activity Description

Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudik, Patrick Shafto
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8096-8108, 2021.

Abstract

We present a novel interactive learning protocol that enables training request-fulfilling agents by verbally describing their activities. Unlike imitation learning (IL), our protocol allows the teaching agent to provide feedback in a language that is most appropriate for them. Compared with reward in reinforcement learning (RL), the description feedback is richer and allows for improved sample complexity. We develop a probabilistic framework and an algorithm that practically implements our protocol. Empirical results in two challenging request-fulfilling problems demonstrate the strengths of our approach: compared with RL baselines, it is more sample-efficient; compared with IL baselines, it achieves competitive success rates without requiring the teaching agent to be able to demonstrate the desired behavior using the learning agent’s actions. Apart from empirical evaluation, we also provide theoretical guarantees for our algorithm under certain assumptions about the teacher and the environment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-nguyen21e, title = {Interactive Learning from Activity Description}, author = {Nguyen, Khanh X and Misra, Dipendra and Schapire, Robert and Dudik, Miroslav and Shafto, Patrick}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {8096--8108}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/nguyen21e/nguyen21e.pdf}, url = {https://proceedings.mlr.press/v139/nguyen21e.html}, abstract = {We present a novel interactive learning protocol that enables training request-fulfilling agents by verbally describing their activities. Unlike imitation learning (IL), our protocol allows the teaching agent to provide feedback in a language that is most appropriate for them. Compared with reward in reinforcement learning (RL), the description feedback is richer and allows for improved sample complexity. We develop a probabilistic framework and an algorithm that practically implements our protocol. Empirical results in two challenging request-fulfilling problems demonstrate the strengths of our approach: compared with RL baselines, it is more sample-efficient; compared with IL baselines, it achieves competitive success rates without requiring the teaching agent to be able to demonstrate the desired behavior using the learning agent’s actions. Apart from empirical evaluation, we also provide theoretical guarantees for our algorithm under certain assumptions about the teacher and the environment.} }
Endnote
%0 Conference Paper %T Interactive Learning from Activity Description %A Khanh X Nguyen %A Dipendra Misra %A Robert Schapire %A Miroslav Dudik %A Patrick Shafto %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-nguyen21e %I PMLR %P 8096--8108 %U https://proceedings.mlr.press/v139/nguyen21e.html %V 139 %X We present a novel interactive learning protocol that enables training request-fulfilling agents by verbally describing their activities. Unlike imitation learning (IL), our protocol allows the teaching agent to provide feedback in a language that is most appropriate for them. Compared with reward in reinforcement learning (RL), the description feedback is richer and allows for improved sample complexity. We develop a probabilistic framework and an algorithm that practically implements our protocol. Empirical results in two challenging request-fulfilling problems demonstrate the strengths of our approach: compared with RL baselines, it is more sample-efficient; compared with IL baselines, it achieves competitive success rates without requiring the teaching agent to be able to demonstrate the desired behavior using the learning agent’s actions. Apart from empirical evaluation, we also provide theoretical guarantees for our algorithm under certain assumptions about the teacher and the environment.
APA
Nguyen, K.X., Misra, D., Schapire, R., Dudik, M. & Shafto, P.. (2021). Interactive Learning from Activity Description. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:8096-8108 Available from https://proceedings.mlr.press/v139/nguyen21e.html.

Related Material