Hierarchically Decoupled Imitation For Morphological Transfer

Donald Hejna, Lerrel Pinto, Pieter Abbeel
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4159-4171, 2020.

Abstract

Learning long-range behaviors on complex high-dimensional agents is a fundamental problem in robot learning. For such tasks, we argue that transferring learned information from a morphologically simpler agent can massively improve the sample efficiency of a more complex one. To this end, we propose a hierarchical decoupling of policies into two parts: an independently learned low-level policy and a transferable high-level policy. To remedy poor transfer performance due to mismatch in morphologies, we contribute two key ideas. First, we show that incentivizing a complex agent’s low-level to imitate a simpler agent’s low-level significantly improves zero-shot high-level transfer. Second, we show that KL-regularized training of the high level stabilizes learning and prevents mode-collapse. Finally, on a suite of publicly released navigation and manipulation environments, we demonstrate the applicability of hierarchical transfer on long-range tasks across morphologies. Our code and videos can be found at https://sites.google.com/berkeley.edu/morphology-transfer.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-hejna20a, title = {Hierarchically Decoupled Imitation For Morphological Transfer}, author = {Hejna, Donald and Pinto, Lerrel and Abbeel, Pieter}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {4159--4171}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/hejna20a/hejna20a.pdf}, url = {https://proceedings.mlr.press/v119/hejna20a.html}, abstract = {Learning long-range behaviors on complex high-dimensional agents is a fundamental problem in robot learning. For such tasks, we argue that transferring learned information from a morphologically simpler agent can massively improve the sample efficiency of a more complex one. To this end, we propose a hierarchical decoupling of policies into two parts: an independently learned low-level policy and a transferable high-level policy. To remedy poor transfer performance due to mismatch in morphologies, we contribute two key ideas. First, we show that incentivizing a complex agent’s low-level to imitate a simpler agent’s low-level significantly improves zero-shot high-level transfer. Second, we show that KL-regularized training of the high level stabilizes learning and prevents mode-collapse. Finally, on a suite of publicly released navigation and manipulation environments, we demonstrate the applicability of hierarchical transfer on long-range tasks across morphologies. Our code and videos can be found at https://sites.google.com/berkeley.edu/morphology-transfer.} }
Endnote
%0 Conference Paper %T Hierarchically Decoupled Imitation For Morphological Transfer %A Donald Hejna %A Lerrel Pinto %A Pieter Abbeel %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-hejna20a %I PMLR %P 4159--4171 %U https://proceedings.mlr.press/v119/hejna20a.html %V 119 %X Learning long-range behaviors on complex high-dimensional agents is a fundamental problem in robot learning. For such tasks, we argue that transferring learned information from a morphologically simpler agent can massively improve the sample efficiency of a more complex one. To this end, we propose a hierarchical decoupling of policies into two parts: an independently learned low-level policy and a transferable high-level policy. To remedy poor transfer performance due to mismatch in morphologies, we contribute two key ideas. First, we show that incentivizing a complex agent’s low-level to imitate a simpler agent’s low-level significantly improves zero-shot high-level transfer. Second, we show that KL-regularized training of the high level stabilizes learning and prevents mode-collapse. Finally, on a suite of publicly released navigation and manipulation environments, we demonstrate the applicability of hierarchical transfer on long-range tasks across morphologies. Our code and videos can be found at https://sites.google.com/berkeley.edu/morphology-transfer.
APA
Hejna, D., Pinto, L. & Abbeel, P.. (2020). Hierarchically Decoupled Imitation For Morphological Transfer. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:4159-4171 Available from https://proceedings.mlr.press/v119/hejna20a.html.

Related Material