Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames

Ondrej Biza, Sjoerd Van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Thomas Kipf
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:2507-2527, 2023.

Abstract

Automatically discovering composable abstractions from raw perceptual data is a long-standing challenge in machine learning. Recent slot-based neural networks that learn about objects in a self-supervised manner have made exciting progress in this direction. However, they typically fall short at adequately capturing spatial symmetries present in the visual world, which leads to sample inefficiency, such as when entangling object appearance and pose. In this paper, we present a simple yet highly effective method for incorporating spatial symmetries via slot-centric reference frames. We incorporate equivariance to per-object pose transformations into the attention and generation mechanism of Slot Attention by translating, scaling, and rotating position encodings. These changes result in little computational overhead, are easy to implement, and can result in large gains in terms of data efficiency and overall improvements to object discovery. We evaluate our method on a wide range of synthetic object discovery benchmarks namely CLEVR, Tetrominoes, CLEVRTex, Objects Room and MultiShapeNet, and show promising improvements on the challenging real-world Waymo Open dataset.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-biza23a, title = {Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames}, author = {Biza, Ondrej and Steenkiste, Sjoerd Van and Sajjadi, Mehdi S. M. and Elsayed, Gamaleldin Fathy and Mahendran, Aravindh and Kipf, Thomas}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {2507--2527}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/biza23a/biza23a.pdf}, url = {https://proceedings.mlr.press/v202/biza23a.html}, abstract = {Automatically discovering composable abstractions from raw perceptual data is a long-standing challenge in machine learning. Recent slot-based neural networks that learn about objects in a self-supervised manner have made exciting progress in this direction. However, they typically fall short at adequately capturing spatial symmetries present in the visual world, which leads to sample inefficiency, such as when entangling object appearance and pose. In this paper, we present a simple yet highly effective method for incorporating spatial symmetries via slot-centric reference frames. We incorporate equivariance to per-object pose transformations into the attention and generation mechanism of Slot Attention by translating, scaling, and rotating position encodings. These changes result in little computational overhead, are easy to implement, and can result in large gains in terms of data efficiency and overall improvements to object discovery. We evaluate our method on a wide range of synthetic object discovery benchmarks namely CLEVR, Tetrominoes, CLEVRTex, Objects Room and MultiShapeNet, and show promising improvements on the challenging real-world Waymo Open dataset.} }
Endnote
%0 Conference Paper %T Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames %A Ondrej Biza %A Sjoerd Van Steenkiste %A Mehdi S. M. Sajjadi %A Gamaleldin Fathy Elsayed %A Aravindh Mahendran %A Thomas Kipf %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-biza23a %I PMLR %P 2507--2527 %U https://proceedings.mlr.press/v202/biza23a.html %V 202 %X Automatically discovering composable abstractions from raw perceptual data is a long-standing challenge in machine learning. Recent slot-based neural networks that learn about objects in a self-supervised manner have made exciting progress in this direction. However, they typically fall short at adequately capturing spatial symmetries present in the visual world, which leads to sample inefficiency, such as when entangling object appearance and pose. In this paper, we present a simple yet highly effective method for incorporating spatial symmetries via slot-centric reference frames. We incorporate equivariance to per-object pose transformations into the attention and generation mechanism of Slot Attention by translating, scaling, and rotating position encodings. These changes result in little computational overhead, are easy to implement, and can result in large gains in terms of data efficiency and overall improvements to object discovery. We evaluate our method on a wide range of synthetic object discovery benchmarks namely CLEVR, Tetrominoes, CLEVRTex, Objects Room and MultiShapeNet, and show promising improvements on the challenging real-world Waymo Open dataset.
APA
Biza, O., Steenkiste, S.V., Sajjadi, M.S.M., Elsayed, G.F., Mahendran, A. & Kipf, T.. (2023). Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:2507-2527 Available from https://proceedings.mlr.press/v202/biza23a.html.

Related Material