Contrastive Inductive Bias Controlling Networks for Reinforcement Learning

Dongxu Li, Shaochen Wang, Kang Chen, Bin Li
Proceedings of The 14th Asian Conference on Machine Learning, PMLR 189:563-578, 2023.

Abstract

Effective learning in an visual-based environment is essential for reinforcement learning (RL) agent, while it has been empirically observed that learning from high dimensional observations such as raw pixels is sample-inefficient. For common practice, RL algorithms for image input often use encoders composed of CNNs to extract useful features from high dimensional observations. Recent studies have shown that CNNs have strong inductive bias towards image styles rather than content (i.e. agent shapes), while content is the information that RL algorithms should focus on. Inspired by this, we suggest reducing the intrinsic style bias of CNNs by proposing Contrastive Inductive Bias Controlling Networks for RL. It can help RL algorithms effectively focus on truly noteworthy information like agents’ own characteristics. Our approach incorporates two transfer networks and feature encoder with contrastive learning methods, guiding RL algorithms to learn more efficiently with sampling. Extensive experiments show that the extended framework greatly enhances the performance of existing model-free methods (i.e. SAC), enabling it to reach state-of-the-art performance on the DeepMind control suite benchmark.

Cite this Paper

BibTeX
@InProceedings{pmlr-v189-li23a, title = {Contrastive Inductive Bias Controlling Networks for Reinforcement Learning}, author = {Li, Dongxu and Wang, Shaochen and Chen, Kang and Li, Bin}, booktitle = {Proceedings of The 14th Asian Conference on Machine Learning}, pages = {563--578}, year = {2023}, editor = {Khan, Emtiyaz and Gonen, Mehmet}, volume = {189}, series = {Proceedings of Machine Learning Research}, month = {12--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v189/li23a/li23a.pdf}, url = {https://proceedings.mlr.press/v189/li23a.html}, abstract = {Effective learning in an visual-based environment is essential for reinforcement learning (RL) agent, while it has been empirically observed that learning from high dimensional observations such as raw pixels is sample-inefficient. For common practice, RL algorithms for image input often use encoders composed of CNNs to extract useful features from high dimensional observations. Recent studies have shown that CNNs have strong inductive bias towards image styles rather than content (i.e. agent shapes), while content is the information that RL algorithms should focus on. Inspired by this, we suggest reducing the intrinsic style bias of CNNs by proposing Contrastive Inductive Bias Controlling Networks for RL. It can help RL algorithms effectively focus on truly noteworthy information like agents’ own characteristics. Our approach incorporates two transfer networks and feature encoder with contrastive learning methods, guiding RL algorithms to learn more efficiently with sampling. Extensive experiments show that the extended framework greatly enhances the performance of existing model-free methods (i.e. SAC), enabling it to reach state-of-the-art performance on the DeepMind control suite benchmark.} }
Endnote
%0 Conference Paper %T Contrastive Inductive Bias Controlling Networks for Reinforcement Learning %A Dongxu Li %A Shaochen Wang %A Kang Chen %A Bin Li %B Proceedings of The 14th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Emtiyaz Khan %E Mehmet Gonen %F pmlr-v189-li23a %I PMLR %P 563--578 %U https://proceedings.mlr.press/v189/li23a.html %V 189 %X Effective learning in an visual-based environment is essential for reinforcement learning (RL) agent, while it has been empirically observed that learning from high dimensional observations such as raw pixels is sample-inefficient. For common practice, RL algorithms for image input often use encoders composed of CNNs to extract useful features from high dimensional observations. Recent studies have shown that CNNs have strong inductive bias towards image styles rather than content (i.e. agent shapes), while content is the information that RL algorithms should focus on. Inspired by this, we suggest reducing the intrinsic style bias of CNNs by proposing Contrastive Inductive Bias Controlling Networks for RL. It can help RL algorithms effectively focus on truly noteworthy information like agents’ own characteristics. Our approach incorporates two transfer networks and feature encoder with contrastive learning methods, guiding RL algorithms to learn more efficiently with sampling. Extensive experiments show that the extended framework greatly enhances the performance of existing model-free methods (i.e. SAC), enabling it to reach state-of-the-art performance on the DeepMind control suite benchmark.
APA
Li, D., Wang, S., Chen, K. & Li, B.. (2023). Contrastive Inductive Bias Controlling Networks for Reinforcement Learning. Proceedings of The 14th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 189:563-578 Available from https://proceedings.mlr.press/v189/li23a.html.

Related Material