Self-supervised Reinforcement Learning with Independently Controllable Subgoals

Andrii Zadaianchuk, Georg Martius, Fanny Yang
Proceedings of the 5th Conference on Robot Learning, PMLR 164:384-394, 2022.

Abstract

To successfully tackle challenging manipulation tasks, autonomous agents must learn a diverse set of skills and how to combine them. Recently, self-supervised agents that set their own abstract goals by exploiting the discovered structure in the environment were shown to perform well on many different tasks. In particular, some of them were applied to learn basic manipulation skills in compositional multi-object environments. However, these methods learn skills without taking the dependencies between objects into account. Thus, the learned skills are difficult to combine in realistic environments. We propose a novel self-supervised agent that estimates relations between environment components and uses them to independently control different parts of the environment state. In addition, the estimated relations between objects can be used to decompose a complex goal into a compatible sequence of subgoals. We show that, by using this framework, an agent can efficiently and automatically learn manipulation tasks in multi-object environments with different relations between objects.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-zadaianchuk22a, title = {Self-supervised Reinforcement Learning with Independently Controllable Subgoals}, author = {Zadaianchuk, Andrii and Martius, Georg and Yang, Fanny}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {384--394}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/zadaianchuk22a/zadaianchuk22a.pdf}, url = {https://proceedings.mlr.press/v164/zadaianchuk22a.html}, abstract = {To successfully tackle challenging manipulation tasks, autonomous agents must learn a diverse set of skills and how to combine them. Recently, self-supervised agents that set their own abstract goals by exploiting the discovered structure in the environment were shown to perform well on many different tasks. In particular, some of them were applied to learn basic manipulation skills in compositional multi-object environments. However, these methods learn skills without taking the dependencies between objects into account. Thus, the learned skills are difficult to combine in realistic environments. We propose a novel self-supervised agent that estimates relations between environment components and uses them to independently control different parts of the environment state. In addition, the estimated relations between objects can be used to decompose a complex goal into a compatible sequence of subgoals. We show that, by using this framework, an agent can efficiently and automatically learn manipulation tasks in multi-object environments with different relations between objects. } }
Endnote
%0 Conference Paper %T Self-supervised Reinforcement Learning with Independently Controllable Subgoals %A Andrii Zadaianchuk %A Georg Martius %A Fanny Yang %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-zadaianchuk22a %I PMLR %P 384--394 %U https://proceedings.mlr.press/v164/zadaianchuk22a.html %V 164 %X To successfully tackle challenging manipulation tasks, autonomous agents must learn a diverse set of skills and how to combine them. Recently, self-supervised agents that set their own abstract goals by exploiting the discovered structure in the environment were shown to perform well on many different tasks. In particular, some of them were applied to learn basic manipulation skills in compositional multi-object environments. However, these methods learn skills without taking the dependencies between objects into account. Thus, the learned skills are difficult to combine in realistic environments. We propose a novel self-supervised agent that estimates relations between environment components and uses them to independently control different parts of the environment state. In addition, the estimated relations between objects can be used to decompose a complex goal into a compatible sequence of subgoals. We show that, by using this framework, an agent can efficiently and automatically learn manipulation tasks in multi-object environments with different relations between objects.
APA
Zadaianchuk, A., Martius, G. & Yang, F.. (2022). Self-supervised Reinforcement Learning with Independently Controllable Subgoals. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:384-394 Available from https://proceedings.mlr.press/v164/zadaianchuk22a.html.

Related Material