Asking for Knowledge (AFK): Training RL Agents to Query External Knowledge Using Language

Iou-Jen Liu, Xingdi Yuan, Marc-Alexandre Côté, Pierre-Yves Oudeyer, Alexander Schwing
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:14073-14093, 2022.

Abstract

To solve difficult tasks, humans ask questions to acquire knowledge from external sources. In contrast, classical reinforcement learning agents lack such an ability and often resort to exploratory behavior. This is exacerbated as few present-day environments support querying for knowledge. In order to study how agents can be taught to query external knowledge via language, we first introduce two new environments: the grid-world-based Q-BabyAI and the text-based Q-TextWorld. In addition to physical interactions, an agent can query an external knowledge source specialized for these environments to gather information. Second, we propose the ‘Asking for Knowledge’ (AFK) agent, which learns to generate language commands to query for meaningful knowledge that helps solve the tasks. AFK leverages a non-parametric memory, a pointer mechanism and an episodic exploration bonus to tackle (1) irrelevant information, (2) a large query language space, (3) delayed reward for making meaningful queries. Extensive experiments demonstrate that the AFK agent outperforms recent baselines on the challenging Q-BabyAI and Q-TextWorld environments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-liu22t, title = {Asking for Knowledge ({AFK}): Training {RL} Agents to Query External Knowledge Using Language}, author = {Liu, Iou-Jen and Yuan, Xingdi and C{\^o}t{\'e}, Marc-Alexandre and Oudeyer, Pierre-Yves and Schwing, Alexander}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {14073--14093}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/liu22t/liu22t.pdf}, url = {https://proceedings.mlr.press/v162/liu22t.html}, abstract = {To solve difficult tasks, humans ask questions to acquire knowledge from external sources. In contrast, classical reinforcement learning agents lack such an ability and often resort to exploratory behavior. This is exacerbated as few present-day environments support querying for knowledge. In order to study how agents can be taught to query external knowledge via language, we first introduce two new environments: the grid-world-based Q-BabyAI and the text-based Q-TextWorld. In addition to physical interactions, an agent can query an external knowledge source specialized for these environments to gather information. Second, we propose the ‘Asking for Knowledge’ (AFK) agent, which learns to generate language commands to query for meaningful knowledge that helps solve the tasks. AFK leverages a non-parametric memory, a pointer mechanism and an episodic exploration bonus to tackle (1) irrelevant information, (2) a large query language space, (3) delayed reward for making meaningful queries. Extensive experiments demonstrate that the AFK agent outperforms recent baselines on the challenging Q-BabyAI and Q-TextWorld environments.} }
Endnote
%0 Conference Paper %T Asking for Knowledge (AFK): Training RL Agents to Query External Knowledge Using Language %A Iou-Jen Liu %A Xingdi Yuan %A Marc-Alexandre Côté %A Pierre-Yves Oudeyer %A Alexander Schwing %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-liu22t %I PMLR %P 14073--14093 %U https://proceedings.mlr.press/v162/liu22t.html %V 162 %X To solve difficult tasks, humans ask questions to acquire knowledge from external sources. In contrast, classical reinforcement learning agents lack such an ability and often resort to exploratory behavior. This is exacerbated as few present-day environments support querying for knowledge. In order to study how agents can be taught to query external knowledge via language, we first introduce two new environments: the grid-world-based Q-BabyAI and the text-based Q-TextWorld. In addition to physical interactions, an agent can query an external knowledge source specialized for these environments to gather information. Second, we propose the ‘Asking for Knowledge’ (AFK) agent, which learns to generate language commands to query for meaningful knowledge that helps solve the tasks. AFK leverages a non-parametric memory, a pointer mechanism and an episodic exploration bonus to tackle (1) irrelevant information, (2) a large query language space, (3) delayed reward for making meaningful queries. Extensive experiments demonstrate that the AFK agent outperforms recent baselines on the challenging Q-BabyAI and Q-TextWorld environments.
APA
Liu, I., Yuan, X., Côté, M., Oudeyer, P. & Schwing, A.. (2022). Asking for Knowledge (AFK): Training RL Agents to Query External Knowledge Using Language. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:14073-14093 Available from https://proceedings.mlr.press/v162/liu22t.html.

Related Material