Roll-Drop: accounting for observation noise with a single parameter

Luigi Campanaro, Daniele De Martini, Siddhant Gangapurwala, Wolfgang Merkt, Ioannis Havoutis
Proceedings of The 5th Annual Learning for Dynamics and Control Conference, PMLR 211:718-730, 2023.

Abstract

This paper proposes a simple strategy for sim-to-real in Deep-Reinforcement Learning (DRL) – called Roll-Drop – that uses dropout during simulation to account for observation noise during deployment without explicitly modelling its distribution for each state. DRL is a promising approach to control robots for highly dynamic and feedback-based manoeuvres, and accurate simulators are crucial to providing cheap and abundant data to learn the desired behaviour. Nevertheless, the simulated data are noiseless and generally show a distributional shift that challenges the deployment on real machines where sensor readings are affected by noise. The standard solution is modelling the latter and injecting it during training; while this requires a thorough system identification, Roll-Drop enhances the robustness to sensor noise by tuning only a single parameter. We demonstrate an 80% success rate when up to 25% noise is injected in the observations, with twice higher robustness than the baselines. We deploy the controller trained in simulation on a Unitree A1 platform and assess this improved robustness on the physical system. Additional resources at: https://sites.google.com/oxfordrobotics.institute/roll-drop

Cite this Paper

BibTeX
@InProceedings{pmlr-v211-campanaro23a, title = {Roll-Drop: accounting for observation noise with a single parameter}, author = {Campanaro, Luigi and Martini, Daniele De and Gangapurwala, Siddhant and Merkt, Wolfgang and Havoutis, Ioannis}, booktitle = {Proceedings of The 5th Annual Learning for Dynamics and Control Conference}, pages = {718--730}, year = {2023}, editor = {Matni, Nikolai and Morari, Manfred and Pappas, George J.}, volume = {211}, series = {Proceedings of Machine Learning Research}, month = {15--16 Jun}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v211/campanaro23a/campanaro23a.pdf}, url = {https://proceedings.mlr.press/v211/campanaro23a.html}, abstract = {This paper proposes a simple strategy for sim-to-real in Deep-Reinforcement Learning (DRL) – called Roll-Drop – that uses dropout during simulation to account for observation noise during deployment without explicitly modelling its distribution for each state. DRL is a promising approach to control robots for highly dynamic and feedback-based manoeuvres, and accurate simulators are crucial to providing cheap and abundant data to learn the desired behaviour. Nevertheless, the simulated data are noiseless and generally show a distributional shift that challenges the deployment on real machines where sensor readings are affected by noise. The standard solution is modelling the latter and injecting it during training; while this requires a thorough system identification, Roll-Drop enhances the robustness to sensor noise by tuning only a single parameter. We demonstrate an 80% success rate when up to 25% noise is injected in the observations, with twice higher robustness than the baselines. We deploy the controller trained in simulation on a Unitree A1 platform and assess this improved robustness on the physical system. Additional resources at: https://sites.google.com/oxfordrobotics.institute/roll-drop} }
Endnote
%0 Conference Paper %T Roll-Drop: accounting for observation noise with a single parameter %A Luigi Campanaro %A Daniele De Martini %A Siddhant Gangapurwala %A Wolfgang Merkt %A Ioannis Havoutis %B Proceedings of The 5th Annual Learning for Dynamics and Control Conference %C Proceedings of Machine Learning Research %D 2023 %E Nikolai Matni %E Manfred Morari %E George J. Pappas %F pmlr-v211-campanaro23a %I PMLR %P 718--730 %U https://proceedings.mlr.press/v211/campanaro23a.html %V 211 %X This paper proposes a simple strategy for sim-to-real in Deep-Reinforcement Learning (DRL) – called Roll-Drop – that uses dropout during simulation to account for observation noise during deployment without explicitly modelling its distribution for each state. DRL is a promising approach to control robots for highly dynamic and feedback-based manoeuvres, and accurate simulators are crucial to providing cheap and abundant data to learn the desired behaviour. Nevertheless, the simulated data are noiseless and generally show a distributional shift that challenges the deployment on real machines where sensor readings are affected by noise. The standard solution is modelling the latter and injecting it during training; while this requires a thorough system identification, Roll-Drop enhances the robustness to sensor noise by tuning only a single parameter. We demonstrate an 80% success rate when up to 25% noise is injected in the observations, with twice higher robustness than the baselines. We deploy the controller trained in simulation on a Unitree A1 platform and assess this improved robustness on the physical system. Additional resources at: https://sites.google.com/oxfordrobotics.institute/roll-drop
APA
Campanaro, L., Martini, D.D., Gangapurwala, S., Merkt, W. & Havoutis, I.. (2023). Roll-Drop: accounting for observation noise with a single parameter. Proceedings of The 5th Annual Learning for Dynamics and Control Conference, in Proceedings of Machine Learning Research 211:718-730 Available from https://proceedings.mlr.press/v211/campanaro23a.html.

Related Material