LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks

Albert Wilcox, Ashwin Balakrishna, Brijen Thananjeyan, Joseph E. Gonzalez, Ken Goldberg
Proceedings of the 5th Conference on Robot Learning, PMLR 164:959-969, 2022.

Abstract

Reinforcement learning (RL) has shown impressive success in exploring high-dimensional environments to learn complex tasks, but can often exhibit unsafe behaviors and require extensive environment interaction when exploration is unconstrained. A promising strategy for learning in dynamically uncertain environments is requiring that the agent can robustly return to learned Safe Sets, where task success (and therefore safety) can be guaranteed. While this approach has been successful in low-dimensions, enforcing this constraint in environments with visual observation spaces is exceedingly challenging. We present a novel continuous representation for Safe Sets framed as a binary classification problem in a learned latent space, which flexibly scales to high-dimensional image observations. We then present a new algorithm, Latent Space Safe Sets (LS3), which uses this representation for long-horizon control. We evaluate LS3 on 4 domains, including a challenging sequential pushing task in simulation and a physical cable routing task. We find that LS3 can use prior task successes to restrict exploration and learn more efficiently than prior algorithms while satisfying constraints. See https://tinyurl.com/latent-safe-sets for supplementary material.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-wilcox22a, title = {LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks}, author = {Wilcox, Albert and Balakrishna, Ashwin and Thananjeyan, Brijen and Gonzalez, Joseph E. and Goldberg, Ken}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {959--969}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/wilcox22a/wilcox22a.pdf}, url = {https://proceedings.mlr.press/v164/wilcox22a.html}, abstract = {Reinforcement learning (RL) has shown impressive success in exploring high-dimensional environments to learn complex tasks, but can often exhibit unsafe behaviors and require extensive environment interaction when exploration is unconstrained. A promising strategy for learning in dynamically uncertain environments is requiring that the agent can robustly return to learned Safe Sets, where task success (and therefore safety) can be guaranteed. While this approach has been successful in low-dimensions, enforcing this constraint in environments with visual observation spaces is exceedingly challenging. We present a novel continuous representation for Safe Sets framed as a binary classification problem in a learned latent space, which flexibly scales to high-dimensional image observations. We then present a new algorithm, Latent Space Safe Sets (LS3), which uses this representation for long-horizon control. We evaluate LS3 on 4 domains, including a challenging sequential pushing task in simulation and a physical cable routing task. We find that LS3 can use prior task successes to restrict exploration and learn more efficiently than prior algorithms while satisfying constraints. See https://tinyurl.com/latent-safe-sets for supplementary material.} }
Endnote
%0 Conference Paper %T LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks %A Albert Wilcox %A Ashwin Balakrishna %A Brijen Thananjeyan %A Joseph E. Gonzalez %A Ken Goldberg %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-wilcox22a %I PMLR %P 959--969 %U https://proceedings.mlr.press/v164/wilcox22a.html %V 164 %X Reinforcement learning (RL) has shown impressive success in exploring high-dimensional environments to learn complex tasks, but can often exhibit unsafe behaviors and require extensive environment interaction when exploration is unconstrained. A promising strategy for learning in dynamically uncertain environments is requiring that the agent can robustly return to learned Safe Sets, where task success (and therefore safety) can be guaranteed. While this approach has been successful in low-dimensions, enforcing this constraint in environments with visual observation spaces is exceedingly challenging. We present a novel continuous representation for Safe Sets framed as a binary classification problem in a learned latent space, which flexibly scales to high-dimensional image observations. We then present a new algorithm, Latent Space Safe Sets (LS3), which uses this representation for long-horizon control. We evaluate LS3 on 4 domains, including a challenging sequential pushing task in simulation and a physical cable routing task. We find that LS3 can use prior task successes to restrict exploration and learn more efficiently than prior algorithms while satisfying constraints. See https://tinyurl.com/latent-safe-sets for supplementary material.
APA
Wilcox, A., Balakrishna, A., Thananjeyan, B., Gonzalez, J.E. & Goldberg, K.. (2022). LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:959-969 Available from https://proceedings.mlr.press/v164/wilcox22a.html.

Related Material