Introspective Action Advising for Interpretable Transfer Learning

Joseph Campbell, Yue Guo, Fiona Xie, Simon Stepputtis, Katia Sycara
Proceedings of The 2nd Conference on Lifelong Learning Agents, PMLR 232:1072-1090, 2023.

Abstract

Transfer learning can be applied in deep reinforcement learning to accelerate the training of a policy in a target task by transferring knowledge from a policy learned in a related source task. This is commonly achieved by copying pretrained weights from the source policy to the target policy prior to training, under the constraint that they use the same model architecture. However, not only does this require a robust representation learned over a wide distribution of states – often failing to transfer between specialist models trained over single tasks – but it is largely uninterpretable and provides little indication of what knowledge is transferred. In this work, we propose an alternative approach to transfer learning between tasks based on action advising, in which a teacher trained in a source task actively guides a student’s exploration in a target task. Through introspection, the teacher is capable of identifying when advice is beneficial to the student and should be given, and when it is not. Our approach allows knowledge transfer between policies agnostic of the underlying representations, and we empirically show that this leads to improved convergence rates in Gridworld and Atari environments while providing insight into what knowledge is transferred.

Cite this Paper

BibTeX
@InProceedings{pmlr-v232-campbell23a, title = {Introspective Action Advising for Interpretable Transfer Learning}, author = {Campbell, Joseph and Guo, Yue and Xie, Fiona and Stepputtis, Simon and Sycara, Katia}, booktitle = {Proceedings of The 2nd Conference on Lifelong Learning Agents}, pages = {1072--1090}, year = {2023}, editor = {Chandar, Sarath and Pascanu, Razvan and Sedghi, Hanie and Precup, Doina}, volume = {232}, series = {Proceedings of Machine Learning Research}, month = {22--25 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v232/campbell23a/campbell23a.pdf}, url = {https://proceedings.mlr.press/v232/campbell23a.html}, abstract = {Transfer learning can be applied in deep reinforcement learning to accelerate the training of a policy in a target task by transferring knowledge from a policy learned in a related source task. This is commonly achieved by copying pretrained weights from the source policy to the target policy prior to training, under the constraint that they use the same model architecture. However, not only does this require a robust representation learned over a wide distribution of states – often failing to transfer between specialist models trained over single tasks – but it is largely uninterpretable and provides little indication of what knowledge is transferred. In this work, we propose an alternative approach to transfer learning between tasks based on action advising, in which a teacher trained in a source task actively guides a student’s exploration in a target task. Through introspection, the teacher is capable of identifying when advice is beneficial to the student and should be given, and when it is not. Our approach allows knowledge transfer between policies agnostic of the underlying representations, and we empirically show that this leads to improved convergence rates in Gridworld and Atari environments while providing insight into what knowledge is transferred. } }
Endnote
%0 Conference Paper %T Introspective Action Advising for Interpretable Transfer Learning %A Joseph Campbell %A Yue Guo %A Fiona Xie %A Simon Stepputtis %A Katia Sycara %B Proceedings of The 2nd Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2023 %E Sarath Chandar %E Razvan Pascanu %E Hanie Sedghi %E Doina Precup %F pmlr-v232-campbell23a %I PMLR %P 1072--1090 %U https://proceedings.mlr.press/v232/campbell23a.html %V 232 %X Transfer learning can be applied in deep reinforcement learning to accelerate the training of a policy in a target task by transferring knowledge from a policy learned in a related source task. This is commonly achieved by copying pretrained weights from the source policy to the target policy prior to training, under the constraint that they use the same model architecture. However, not only does this require a robust representation learned over a wide distribution of states – often failing to transfer between specialist models trained over single tasks – but it is largely uninterpretable and provides little indication of what knowledge is transferred. In this work, we propose an alternative approach to transfer learning between tasks based on action advising, in which a teacher trained in a source task actively guides a student’s exploration in a target task. Through introspection, the teacher is capable of identifying when advice is beneficial to the student and should be given, and when it is not. Our approach allows knowledge transfer between policies agnostic of the underlying representations, and we empirically show that this leads to improved convergence rates in Gridworld and Atari environments while providing insight into what knowledge is transferred.
APA
Campbell, J., Guo, Y., Xie, F., Stepputtis, S. & Sycara, K.. (2023). Introspective Action Advising for Interpretable Transfer Learning. Proceedings of The 2nd Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 232:1072-1090 Available from https://proceedings.mlr.press/v232/campbell23a.html.

Related Material