Haptics-based Curiosity for Sparse-reward Tasks

Sai Rajeswar, Cyril Ibrahim, Nitin Surya, Florian Golemo, David Vazquez, Aaron Courville, Pedro O. Pinheiro
Proceedings of the 5th Conference on Robot Learning, PMLR 164:395-405, 2022.

Abstract

Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to “feel" like. We encourage exploration by rewarding interactions where the expectation and the experience do not match. We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and find that our method performs better and is more sample-efficient.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-rajeswar22a, title = {Haptics-based Curiosity for Sparse-reward Tasks}, author = {Rajeswar, Sai and Ibrahim, Cyril and Surya, Nitin and Golemo, Florian and Vazquez, David and Courville, Aaron and Pinheiro, Pedro O.}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {395--405}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/rajeswar22a/rajeswar22a.pdf}, url = {https://proceedings.mlr.press/v164/rajeswar22a.html}, abstract = {Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to “feel" like. We encourage exploration by rewarding interactions where the expectation and the experience do not match. We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and find that our method performs better and is more sample-efficient.} }
Endnote
%0 Conference Paper %T Haptics-based Curiosity for Sparse-reward Tasks %A Sai Rajeswar %A Cyril Ibrahim %A Nitin Surya %A Florian Golemo %A David Vazquez %A Aaron Courville %A Pedro O. Pinheiro %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-rajeswar22a %I PMLR %P 395--405 %U https://proceedings.mlr.press/v164/rajeswar22a.html %V 164 %X Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to “feel" like. We encourage exploration by rewarding interactions where the expectation and the experience do not match. We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and find that our method performs better and is more sample-efficient.
APA
Rajeswar, S., Ibrahim, C., Surya, N., Golemo, F., Vazquez, D., Courville, A. & Pinheiro, P.O.. (2022). Haptics-based Curiosity for Sparse-reward Tasks. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:395-405 Available from https://proceedings.mlr.press/v164/rajeswar22a.html.

Related Material