Structured World Belief for Reinforcement Learning in POMDP

Gautam Singh, Skand Peri, Junghyun Kim, Hyunseok Kim, Sungjin Ahn
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9744-9755, 2021.

Abstract

Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments due to the lack of belief states. In this paper, we propose Structured World Belief, a model for learning and inference of object-centric belief states. Inferred by Sequential Monte Carlo (SMC), our belief states provide multiple object-centric scene hypotheses. To synergize the benefits of SMC particles with object representations, we also propose a new object-centric dynamics model that considers the inductive bias of object permanence. This enables tracking of object states even when they are invisible for a long time. To further facilitate object tracking in this regime, we allow our model to attend flexibly to any spatial location in the image which was restricted in previous models. In experiments, we show that object-centric belief provides a more accurate and robust performance for filtering and generation. Furthermore, we show the efficacy of structured world belief in improving the performance of reinforcement learning, planning and supervised reasoning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-singh21a, title = {Structured World Belief for Reinforcement Learning in POMDP}, author = {Singh, Gautam and Peri, Skand and Kim, Junghyun and Kim, Hyunseok and Ahn, Sungjin}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9744--9755}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/singh21a/singh21a.pdf}, url = {https://proceedings.mlr.press/v139/singh21a.html}, abstract = {Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments due to the lack of belief states. In this paper, we propose Structured World Belief, a model for learning and inference of object-centric belief states. Inferred by Sequential Monte Carlo (SMC), our belief states provide multiple object-centric scene hypotheses. To synergize the benefits of SMC particles with object representations, we also propose a new object-centric dynamics model that considers the inductive bias of object permanence. This enables tracking of object states even when they are invisible for a long time. To further facilitate object tracking in this regime, we allow our model to attend flexibly to any spatial location in the image which was restricted in previous models. In experiments, we show that object-centric belief provides a more accurate and robust performance for filtering and generation. Furthermore, we show the efficacy of structured world belief in improving the performance of reinforcement learning, planning and supervised reasoning.} }
Endnote
%0 Conference Paper %T Structured World Belief for Reinforcement Learning in POMDP %A Gautam Singh %A Skand Peri %A Junghyun Kim %A Hyunseok Kim %A Sungjin Ahn %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-singh21a %I PMLR %P 9744--9755 %U https://proceedings.mlr.press/v139/singh21a.html %V 139 %X Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments due to the lack of belief states. In this paper, we propose Structured World Belief, a model for learning and inference of object-centric belief states. Inferred by Sequential Monte Carlo (SMC), our belief states provide multiple object-centric scene hypotheses. To synergize the benefits of SMC particles with object representations, we also propose a new object-centric dynamics model that considers the inductive bias of object permanence. This enables tracking of object states even when they are invisible for a long time. To further facilitate object tracking in this regime, we allow our model to attend flexibly to any spatial location in the image which was restricted in previous models. In experiments, we show that object-centric belief provides a more accurate and robust performance for filtering and generation. Furthermore, we show the efficacy of structured world belief in improving the performance of reinforcement learning, planning and supervised reasoning.
APA
Singh, G., Peri, S., Kim, J., Kim, H. & Ahn, S.. (2021). Structured World Belief for Reinforcement Learning in POMDP. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9744-9755 Available from https://proceedings.mlr.press/v139/singh21a.html.

Related Material