Relational Object-Centric Actor-Critic

Leonid Anatolievich Ugadiarov, Vitaliy Vorobyov, Aleksandr Panov
Proceedings of the Fourth Conference on Causal Learning and Reasoning, PMLR 275:1450-1476, 2025.

Abstract

There have recently been significant advances in the problem of unsupervised object-centric representation learning and its application to downstream tasks. The latest works support the argument that employing disentangled object representations in image-based object-centric reinforcement learning tasks facilitates policy learning. We propose a novel object-centric reinforcement learning algorithm combining actor-critic and model-based approaches, by incorporating an object-centric world model in critic. The proposed method fills a research gap in developing efficient object-centric world models for reinforcement learning settings that can be used for environments with discrete or continuous action spaces. We evaluated our algorithm in simulated 3D robotic environment and a 2D environment with compositional structure. As baselines, we consider the state-of-the-art model-free actor-critic algorithm built upon transformer architecture and the state-of-the-art monolithic model-based algorithm. While the proposed method demonstrates comparable performance to the baselines in easier tasks, it outperforms the baselines within the 1M environment step budget in more challenging tasks increased number of objects or more complex dynamics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v275-ugadiarov25a, title = {Relational Object-Centric Actor-Critic}, author = {Ugadiarov, Leonid Anatolievich and Vorobyov, Vitaliy and Panov, Aleksandr}, booktitle = {Proceedings of the Fourth Conference on Causal Learning and Reasoning}, pages = {1450--1476}, year = {2025}, editor = {Huang, Biwei and Drton, Mathias}, volume = {275}, series = {Proceedings of Machine Learning Research}, month = {07--09 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v275/main/assets/ugadiarov25a/ugadiarov25a.pdf}, url = {https://proceedings.mlr.press/v275/ugadiarov25a.html}, abstract = {There have recently been significant advances in the problem of unsupervised object-centric representation learning and its application to downstream tasks. The latest works support the argument that employing disentangled object representations in image-based object-centric reinforcement learning tasks facilitates policy learning. We propose a novel object-centric reinforcement learning algorithm combining actor-critic and model-based approaches, by incorporating an object-centric world model in critic. The proposed method fills a research gap in developing efficient object-centric world models for reinforcement learning settings that can be used for environments with discrete or continuous action spaces. We evaluated our algorithm in simulated 3D robotic environment and a 2D environment with compositional structure. As baselines, we consider the state-of-the-art model-free actor-critic algorithm built upon transformer architecture and the state-of-the-art monolithic model-based algorithm. While the proposed method demonstrates comparable performance to the baselines in easier tasks, it outperforms the baselines within the 1M environment step budget in more challenging tasks increased number of objects or more complex dynamics.} }
Endnote
%0 Conference Paper %T Relational Object-Centric Actor-Critic %A Leonid Anatolievich Ugadiarov %A Vitaliy Vorobyov %A Aleksandr Panov %B Proceedings of the Fourth Conference on Causal Learning and Reasoning %C Proceedings of Machine Learning Research %D 2025 %E Biwei Huang %E Mathias Drton %F pmlr-v275-ugadiarov25a %I PMLR %P 1450--1476 %U https://proceedings.mlr.press/v275/ugadiarov25a.html %V 275 %X There have recently been significant advances in the problem of unsupervised object-centric representation learning and its application to downstream tasks. The latest works support the argument that employing disentangled object representations in image-based object-centric reinforcement learning tasks facilitates policy learning. We propose a novel object-centric reinforcement learning algorithm combining actor-critic and model-based approaches, by incorporating an object-centric world model in critic. The proposed method fills a research gap in developing efficient object-centric world models for reinforcement learning settings that can be used for environments with discrete or continuous action spaces. We evaluated our algorithm in simulated 3D robotic environment and a 2D environment with compositional structure. As baselines, we consider the state-of-the-art model-free actor-critic algorithm built upon transformer architecture and the state-of-the-art monolithic model-based algorithm. While the proposed method demonstrates comparable performance to the baselines in easier tasks, it outperforms the baselines within the 1M environment step budget in more challenging tasks increased number of objects or more complex dynamics.
APA
Ugadiarov, L.A., Vorobyov, V. & Panov, A.. (2025). Relational Object-Centric Actor-Critic. Proceedings of the Fourth Conference on Causal Learning and Reasoning, in Proceedings of Machine Learning Research 275:1450-1476 Available from https://proceedings.mlr.press/v275/ugadiarov25a.html.

Related Material