Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real

Zhaoming Xie, Patrick Clary, Jeremy Dao, Pedro Morais, Jonanthan Hurst, Michiel Panne
Proceedings of the Conference on Robot Learning, PMLR 100:317-329, 2020.

Abstract

Deep reinforcement learning (DRL) is a promising approach for developing legged locomotion skills. However, current work commonly describes DRL as being a one-shot process, where the state, action and reward are assumed to be well defined and are directly used by an RL algorithm to obtain policies. In this paper, we describe and document an iterative design approach, which reflects the multiple design iterations of the reward that are often (if not always) needed in practice. Throughout the process, transfer learning is achieved via Deterministic Action Stochastic State (DASS) tuples, representing the deterministic policy actions associated with states visited by the stochastic policy. We demonstrate the transfer of policies learned in simulation to the physical robot without dynamics randomization. We also identify several key components that are critical for sim-to-real transfer in our setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v100-xie20a, title = {Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real}, author = {Xie, Zhaoming and Clary, Patrick and Dao, Jeremy and Morais, Pedro and Hurst, Jonanthan and van de Panne, Michiel}, booktitle = {Proceedings of the Conference on Robot Learning}, pages = {317--329}, year = {2020}, editor = {Kaelbling, Leslie Pack and Kragic, Danica and Sugiura, Komei}, volume = {100}, series = {Proceedings of Machine Learning Research}, month = {30 Oct--01 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v100/xie20a/xie20a.pdf}, url = {https://proceedings.mlr.press/v100/xie20a.html}, abstract = {Deep reinforcement learning (DRL) is a promising approach for developing legged locomotion skills. However, current work commonly describes DRL as being a one-shot process, where the state, action and reward are assumed to be well defined and are directly used by an RL algorithm to obtain policies. In this paper, we describe and document an iterative design approach, which reflects the multiple design iterations of the reward that are often (if not always) needed in practice. Throughout the process, transfer learning is achieved via Deterministic Action Stochastic State (DASS) tuples, representing the deterministic policy actions associated with states visited by the stochastic policy. We demonstrate the transfer of policies learned in simulation to the physical robot without dynamics randomization. We also identify several key components that are critical for sim-to-real transfer in our setting.} }
Endnote
%0 Conference Paper %T Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real %A Zhaoming Xie %A Patrick Clary %A Jeremy Dao %A Pedro Morais %A Jonanthan Hurst %A Michiel Panne %B Proceedings of the Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2020 %E Leslie Pack Kaelbling %E Danica Kragic %E Komei Sugiura %F pmlr-v100-xie20a %I PMLR %P 317--329 %U https://proceedings.mlr.press/v100/xie20a.html %V 100 %X Deep reinforcement learning (DRL) is a promising approach for developing legged locomotion skills. However, current work commonly describes DRL as being a one-shot process, where the state, action and reward are assumed to be well defined and are directly used by an RL algorithm to obtain policies. In this paper, we describe and document an iterative design approach, which reflects the multiple design iterations of the reward that are often (if not always) needed in practice. Throughout the process, transfer learning is achieved via Deterministic Action Stochastic State (DASS) tuples, representing the deterministic policy actions associated with states visited by the stochastic policy. We demonstrate the transfer of policies learned in simulation to the physical robot without dynamics randomization. We also identify several key components that are critical for sim-to-real transfer in our setting.
APA
Xie, Z., Clary, P., Dao, J., Morais, P., Hurst, J. & Panne, M.. (2020). Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real. Proceedings of the Conference on Robot Learning, in Proceedings of Machine Learning Research 100:317-329 Available from https://proceedings.mlr.press/v100/xie20a.html.

Related Material