Object Permanence Emerges in a Random Walk along Memory

Pavel Tokmakov, Allan Jabri, Jie Li, Adrien Gaidon
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:21506-21519, 2022.

Abstract

This paper proposes a self-supervised objective for learning representations that localize objects under occlusion - a property known as object permanence. A central question is the choice of learning signal in cases of total occlusion. Rather than directly supervising the locations of invisible objects, we propose a self-supervised objective that requires neither human annotation, nor assumptions about object dynamics. We show that object permanence can emerge by optimizing for temporal coherence of memory: we fit a Markov walk along a space-time graph of memories, where the states in each time step are non-Markovian features from a sequence encoder. This leads to a memory representation that stores occluded objects and predicts their motion, to better localize them. The resulting model outperforms existing approaches on several datasets of increasing complexity and realism, despite requiring minimal supervision, and hence being broadly applicable.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-tokmakov22a, title = {Object Permanence Emerges in a Random Walk along Memory}, author = {Tokmakov, Pavel and Jabri, Allan and Li, Jie and Gaidon, Adrien}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {21506--21519}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/tokmakov22a/tokmakov22a.pdf}, url = {https://proceedings.mlr.press/v162/tokmakov22a.html}, abstract = {This paper proposes a self-supervised objective for learning representations that localize objects under occlusion - a property known as object permanence. A central question is the choice of learning signal in cases of total occlusion. Rather than directly supervising the locations of invisible objects, we propose a self-supervised objective that requires neither human annotation, nor assumptions about object dynamics. We show that object permanence can emerge by optimizing for temporal coherence of memory: we fit a Markov walk along a space-time graph of memories, where the states in each time step are non-Markovian features from a sequence encoder. This leads to a memory representation that stores occluded objects and predicts their motion, to better localize them. The resulting model outperforms existing approaches on several datasets of increasing complexity and realism, despite requiring minimal supervision, and hence being broadly applicable.} }
Endnote
%0 Conference Paper %T Object Permanence Emerges in a Random Walk along Memory %A Pavel Tokmakov %A Allan Jabri %A Jie Li %A Adrien Gaidon %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-tokmakov22a %I PMLR %P 21506--21519 %U https://proceedings.mlr.press/v162/tokmakov22a.html %V 162 %X This paper proposes a self-supervised objective for learning representations that localize objects under occlusion - a property known as object permanence. A central question is the choice of learning signal in cases of total occlusion. Rather than directly supervising the locations of invisible objects, we propose a self-supervised objective that requires neither human annotation, nor assumptions about object dynamics. We show that object permanence can emerge by optimizing for temporal coherence of memory: we fit a Markov walk along a space-time graph of memories, where the states in each time step are non-Markovian features from a sequence encoder. This leads to a memory representation that stores occluded objects and predicts their motion, to better localize them. The resulting model outperforms existing approaches on several datasets of increasing complexity and realism, despite requiring minimal supervision, and hence being broadly applicable.
APA
Tokmakov, P., Jabri, A., Li, J. & Gaidon, A.. (2022). Object Permanence Emerges in a Random Walk along Memory. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:21506-21519 Available from https://proceedings.mlr.press/v162/tokmakov22a.html.

Related Material