Action Space Design in Reinforcement Learning for Robot Motor Skills

Julian Eßer, Gabriel B. Margolis, Oliver Urbann, Sören Kerner, Pulkit Agrawal
Proceedings of The 8th Conference on Robot Learning, PMLR 270:4021-4032, 2025.

Abstract

Practitioners often rely on intuition to select action spaces for learning. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled-legged robot, a quadruped robot, and a simulated suite of locomotion, manipulation, and control tasks. We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially in a task-dependent way. Moreover, we find that much of the practical impact of action space selection on learning dynamics can be explained by improved policy initialization and behavior between timesteps.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-esser25a, title = {Action Space Design in Reinforcement Learning for Robot Motor Skills}, author = {E\ss{}er, Julian and Margolis, Gabriel B. and Urbann, Oliver and Kerner, S{\"{o}}ren and Agrawal, Pulkit}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {4021--4032}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/esser25a/esser25a.pdf}, url = {https://proceedings.mlr.press/v270/esser25a.html}, abstract = {Practitioners often rely on intuition to select action spaces for learning. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled-legged robot, a quadruped robot, and a simulated suite of locomotion, manipulation, and control tasks. We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially in a task-dependent way. Moreover, we find that much of the practical impact of action space selection on learning dynamics can be explained by improved policy initialization and behavior between timesteps.} }
Endnote
%0 Conference Paper %T Action Space Design in Reinforcement Learning for Robot Motor Skills %A Julian Eßer %A Gabriel B. Margolis %A Oliver Urbann %A Sören Kerner %A Pulkit Agrawal %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-esser25a %I PMLR %P 4021--4032 %U https://proceedings.mlr.press/v270/esser25a.html %V 270 %X Practitioners often rely on intuition to select action spaces for learning. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled-legged robot, a quadruped robot, and a simulated suite of locomotion, manipulation, and control tasks. We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially in a task-dependent way. Moreover, we find that much of the practical impact of action space selection on learning dynamics can be explained by improved policy initialization and behavior between timesteps.
APA
Eßer, J., Margolis, G.B., Urbann, O., Kerner, S. & Agrawal, P.. (2025). Action Space Design in Reinforcement Learning for Robot Motor Skills. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:4021-4032 Available from https://proceedings.mlr.press/v270/esser25a.html.

Related Material