When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions

Zhening Li, Gabriel Poesia, Armando Solar-Lezama
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:28568-28596, 2024.

Abstract

Skills are temporal abstractions that are intended to improve reinforcement learning (RL) performance through hierarchical RL. Despite our intuition about the properties of an environment that make skills useful, a precise characterization has been absent. We provide the first such characterization, focusing on the utility of deterministic skills in deterministic sparse-reward environments with finite action spaces. We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible. Additional theoretical results suggest that skills benefit exploration more than they benefit learning from existing experience, and that using unexpressive skills such as macroactions may worsen RL performance. We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-li24be, title = {When Do Skills Help Reinforcement Learning? {A} Theoretical Analysis of Temporal Abstractions}, author = {Li, Zhening and Poesia, Gabriel and Solar-Lezama, Armando}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {28568--28596}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/li24be/li24be.pdf}, url = {https://proceedings.mlr.press/v235/li24be.html}, abstract = {Skills are temporal abstractions that are intended to improve reinforcement learning (RL) performance through hierarchical RL. Despite our intuition about the properties of an environment that make skills useful, a precise characterization has been absent. We provide the first such characterization, focusing on the utility of deterministic skills in deterministic sparse-reward environments with finite action spaces. We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible. Additional theoretical results suggest that skills benefit exploration more than they benefit learning from existing experience, and that using unexpressive skills such as macroactions may worsen RL performance. We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.} }
Endnote
%0 Conference Paper %T When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions %A Zhening Li %A Gabriel Poesia %A Armando Solar-Lezama %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-li24be %I PMLR %P 28568--28596 %U https://proceedings.mlr.press/v235/li24be.html %V 235 %X Skills are temporal abstractions that are intended to improve reinforcement learning (RL) performance through hierarchical RL. Despite our intuition about the properties of an environment that make skills useful, a precise characterization has been absent. We provide the first such characterization, focusing on the utility of deterministic skills in deterministic sparse-reward environments with finite action spaces. We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible. Additional theoretical results suggest that skills benefit exploration more than they benefit learning from existing experience, and that using unexpressive skills such as macroactions may worsen RL performance. We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.
APA
Li, Z., Poesia, G. & Solar-Lezama, A.. (2024). When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:28568-28596 Available from https://proceedings.mlr.press/v235/li24be.html.

Related Material