A User’s Guide to Calibrating Robotic Simulators

Bhairav Mehta, Ankur Handa, Dieter Fox, Fabio Ramos
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:1326-1340, 2021.

Abstract

Simulators are a critical component of modern robotics research. Strategies for both perception and decision making can be studied in simulation first, before deployed to real world systems, saving on time and costs. Despite significant progress on the development of sim-to-real algorithms, the analysis of different methods is still conducted in an ad-hoc manner without a consistent set of tests and metrics for comparison. This paper intends to fill this gap and proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterise and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behaviour and main properties of sim-to-real algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-mehta21a, title = {A User’s Guide to Calibrating Robotic Simulators}, author = {Mehta, Bhairav and Handa, Ankur and Fox, Dieter and Ramos, Fabio}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {1326--1340}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/mehta21a/mehta21a.pdf}, url = {https://proceedings.mlr.press/v155/mehta21a.html}, abstract = {Simulators are a critical component of modern robotics research. Strategies for both perception and decision making can be studied in simulation first, before deployed to real world systems, saving on time and costs. Despite significant progress on the development of sim-to-real algorithms, the analysis of different methods is still conducted in an ad-hoc manner without a consistent set of tests and metrics for comparison. This paper intends to fill this gap and proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterise and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behaviour and main properties of sim-to-real algorithms.} }
Endnote
%0 Conference Paper %T A User’s Guide to Calibrating Robotic Simulators %A Bhairav Mehta %A Ankur Handa %A Dieter Fox %A Fabio Ramos %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-mehta21a %I PMLR %P 1326--1340 %U https://proceedings.mlr.press/v155/mehta21a.html %V 155 %X Simulators are a critical component of modern robotics research. Strategies for both perception and decision making can be studied in simulation first, before deployed to real world systems, saving on time and costs. Despite significant progress on the development of sim-to-real algorithms, the analysis of different methods is still conducted in an ad-hoc manner without a consistent set of tests and metrics for comparison. This paper intends to fill this gap and proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterise and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behaviour and main properties of sim-to-real algorithms.
APA
Mehta, B., Handa, A., Fox, D. & Ramos, F.. (2021). A User’s Guide to Calibrating Robotic Simulators. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:1326-1340 Available from https://proceedings.mlr.press/v155/mehta21a.html.

Related Material