Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence

Zhiyuan Li, Rongzhen Zhao, Wenyan Yang, Wenshuai Zhao, Pekka Marttinen, Joni Pajarinen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:72684-72705, 2026.

Abstract

The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-li26js, title = {Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence}, author = {Li, Zhiyuan and Zhao, Rongzhen and Yang, Wenyan and Zhao, Wenshuai and Marttinen, Pekka and Pajarinen, Joni}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {72684--72705}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26js/li26js.pdf}, url = {https://proceedings.mlr.press/v306/li26js.html}, abstract = {The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/} }
Endnote
%0 Conference Paper %T Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence %A Zhiyuan Li %A Rongzhen Zhao %A Wenyan Yang %A Wenshuai Zhao %A Pekka Marttinen %A Joni Pajarinen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26js %I PMLR %P 72684--72705 %U https://proceedings.mlr.press/v306/li26js.html %V 306 %X The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/
APA
Li, Z., Zhao, R., Yang, W., Zhao, W., Marttinen, P. & Pajarinen, J.. (2026). Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:72684-72705 Available from https://proceedings.mlr.press/v306/li26js.html.

Related Material