Equivariant $Q$ Learning in Spatial Action Spaces

Dian Wang, Robin Walters, Xupeng Zhu, Robert Platt
Proceedings of the 5th Conference on Robot Learning, PMLR 164:1713-1723, 2022.

Abstract

Recently, a variety of new equivariant neural network model architectures have been proposed that generalize better over rotational and reflectional symmetries than standard models. These models are relevant to robotics because many robotics problems can be expressed in a rotationally symmetric way. This paper focuses on equivariance over a visual state space and a spatial action space – the setting where the robot action space includes a subset of $\rm{SE}(2)$. In this situation, we know a priori that rotations and translations in the state image should result in the same rotations and translations in the spatial action dimensions of the optimal policy. Therefore, we can use equivariant model architectures to make $Q$ learning more sample efficient. This paper identifies when the optimal $Q$ function is equivariant and proposes $Q$ network architectures for this setting. We show experimentally that this approach outperforms standard methods in a set of challenging manipulation problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-wang22j, title = {Equivariant $Q$ Learning in Spatial Action Spaces}, author = {Wang, Dian and Walters, Robin and Zhu, Xupeng and Platt, Robert}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {1713--1723}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/wang22j/wang22j.pdf}, url = {https://proceedings.mlr.press/v164/wang22j.html}, abstract = {Recently, a variety of new equivariant neural network model architectures have been proposed that generalize better over rotational and reflectional symmetries than standard models. These models are relevant to robotics because many robotics problems can be expressed in a rotationally symmetric way. This paper focuses on equivariance over a visual state space and a spatial action space – the setting where the robot action space includes a subset of $\rm{SE}(2)$. In this situation, we know a priori that rotations and translations in the state image should result in the same rotations and translations in the spatial action dimensions of the optimal policy. Therefore, we can use equivariant model architectures to make $Q$ learning more sample efficient. This paper identifies when the optimal $Q$ function is equivariant and proposes $Q$ network architectures for this setting. We show experimentally that this approach outperforms standard methods in a set of challenging manipulation problems. } }
Endnote
%0 Conference Paper %T Equivariant $Q$ Learning in Spatial Action Spaces %A Dian Wang %A Robin Walters %A Xupeng Zhu %A Robert Platt %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-wang22j %I PMLR %P 1713--1723 %U https://proceedings.mlr.press/v164/wang22j.html %V 164 %X Recently, a variety of new equivariant neural network model architectures have been proposed that generalize better over rotational and reflectional symmetries than standard models. These models are relevant to robotics because many robotics problems can be expressed in a rotationally symmetric way. This paper focuses on equivariance over a visual state space and a spatial action space – the setting where the robot action space includes a subset of $\rm{SE}(2)$. In this situation, we know a priori that rotations and translations in the state image should result in the same rotations and translations in the spatial action dimensions of the optimal policy. Therefore, we can use equivariant model architectures to make $Q$ learning more sample efficient. This paper identifies when the optimal $Q$ function is equivariant and proposes $Q$ network architectures for this setting. We show experimentally that this approach outperforms standard methods in a set of challenging manipulation problems.
APA
Wang, D., Walters, R., Zhu, X. & Platt, R.. (2022). Equivariant $Q$ Learning in Spatial Action Spaces. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:1713-1723 Available from https://proceedings.mlr.press/v164/wang22j.html.

Related Material