Representation Learning in Deep RL via Discrete Information Bottleneck

Riashat Islam, Hongyu Zang, Manan Tomar, Aniket Didolkar, Md Mofijul Islam, Samin Yeasar Arnob, Tariq Iqbal, Xin Li, Anirudh Goyal, Nicolas Heess, Alex Lamb
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:8699-8722, 2023.

Abstract

Several self-supervised representation learning methods have been proposed for reinforcement learning (RL) with rich observations. For real world applications of RL, recovering underlying latent states is crucial, particularly when sensory inputs can contain irrelevant and exogenous information. In this work, we study how information bottlenecjs can be used to construct latent states efficiently in the presence of task irrelevant information. We propose architectures that utilize variational and discrete information bottleneck, coined as RepDIB, to learn structured factorized representations. Exploiting the expressiveness bought by factorized representations, we introduce a simple, yet effective, bottleneck that can be integrated with any existing self supervised objective for RL. We demonstrate this across several online and offline RL benchmarks, along with a real robot arm task, where we find that compressed representations with RepDIB can lead to strong performance improvements, as the learnt bottlenecks can help predict only the relevant state, while ignoring irrelevant information.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-islam23a, title = {Representation Learning in Deep RL via Discrete Information Bottleneck}, author = {Islam, Riashat and Zang, Hongyu and Tomar, Manan and Didolkar, Aniket and Islam, Md Mofijul and Arnob, Samin Yeasar and Iqbal, Tariq and Li, Xin and Goyal, Anirudh and Heess, Nicolas and Lamb, Alex}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {8699--8722}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/islam23a/islam23a.pdf}, url = {https://proceedings.mlr.press/v206/islam23a.html}, abstract = {Several self-supervised representation learning methods have been proposed for reinforcement learning (RL) with rich observations. For real world applications of RL, recovering underlying latent states is crucial, particularly when sensory inputs can contain irrelevant and exogenous information. In this work, we study how information bottlenecjs can be used to construct latent states efficiently in the presence of task irrelevant information. We propose architectures that utilize variational and discrete information bottleneck, coined as RepDIB, to learn structured factorized representations. Exploiting the expressiveness bought by factorized representations, we introduce a simple, yet effective, bottleneck that can be integrated with any existing self supervised objective for RL. We demonstrate this across several online and offline RL benchmarks, along with a real robot arm task, where we find that compressed representations with RepDIB can lead to strong performance improvements, as the learnt bottlenecks can help predict only the relevant state, while ignoring irrelevant information.} }
Endnote
%0 Conference Paper %T Representation Learning in Deep RL via Discrete Information Bottleneck %A Riashat Islam %A Hongyu Zang %A Manan Tomar %A Aniket Didolkar %A Md Mofijul Islam %A Samin Yeasar Arnob %A Tariq Iqbal %A Xin Li %A Anirudh Goyal %A Nicolas Heess %A Alex Lamb %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-islam23a %I PMLR %P 8699--8722 %U https://proceedings.mlr.press/v206/islam23a.html %V 206 %X Several self-supervised representation learning methods have been proposed for reinforcement learning (RL) with rich observations. For real world applications of RL, recovering underlying latent states is crucial, particularly when sensory inputs can contain irrelevant and exogenous information. In this work, we study how information bottlenecjs can be used to construct latent states efficiently in the presence of task irrelevant information. We propose architectures that utilize variational and discrete information bottleneck, coined as RepDIB, to learn structured factorized representations. Exploiting the expressiveness bought by factorized representations, we introduce a simple, yet effective, bottleneck that can be integrated with any existing self supervised objective for RL. We demonstrate this across several online and offline RL benchmarks, along with a real robot arm task, where we find that compressed representations with RepDIB can lead to strong performance improvements, as the learnt bottlenecks can help predict only the relevant state, while ignoring irrelevant information.
APA
Islam, R., Zang, H., Tomar, M., Didolkar, A., Islam, M.M., Arnob, S.Y., Iqbal, T., Li, X., Goyal, A., Heess, N. & Lamb, A.. (2023). Representation Learning in Deep RL via Discrete Information Bottleneck. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:8699-8722 Available from https://proceedings.mlr.press/v206/islam23a.html.

Related Material