Hierarchical Representation Learning for Markov Decision Processes

Lorenzo Steccanella, Simone Totaro, Anders Jonsson
Proceedings of The 2nd Conference on Lifelong Learning Agents, PMLR 232:568-585, 2023.

Abstract

In this paper, we present a novel method for learning reward-agnostic hierarchical representations of Markov Decision Processes. Our method works by partitioning the state space into subsets, and defines subtasks for performing transitions between the partitions. At the high level, we use model-based planning to decide which subtask to pursue next from a given partition. We formulate the problem of partitioning the state space as an optimization problem that can be solved using gradient descent given a set of sampled trajectories, making our method suitable for high-dimensional problems with large state spaces. We empirically validate the method, by showing that it can successfully learn useful hierarchical representations in domains with high-dimensional states. Once learned, the hierarchical representation can be used to solve different tasks in the given domain, thus generalizing knowledge across tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v232-steccanella23a, title = {Hierarchical Representation Learning for Markov Decision Processes}, author = {Steccanella, Lorenzo and Totaro, Simone and Jonsson, Anders}, booktitle = {Proceedings of The 2nd Conference on Lifelong Learning Agents}, pages = {568--585}, year = {2023}, editor = {Chandar, Sarath and Pascanu, Razvan and Sedghi, Hanie and Precup, Doina}, volume = {232}, series = {Proceedings of Machine Learning Research}, month = {22--25 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v232/steccanella23a/steccanella23a.pdf}, url = {https://proceedings.mlr.press/v232/steccanella23a.html}, abstract = {In this paper, we present a novel method for learning reward-agnostic hierarchical representations of Markov Decision Processes. Our method works by partitioning the state space into subsets, and defines subtasks for performing transitions between the partitions. At the high level, we use model-based planning to decide which subtask to pursue next from a given partition. We formulate the problem of partitioning the state space as an optimization problem that can be solved using gradient descent given a set of sampled trajectories, making our method suitable for high-dimensional problems with large state spaces. We empirically validate the method, by showing that it can successfully learn useful hierarchical representations in domains with high-dimensional states. Once learned, the hierarchical representation can be used to solve different tasks in the given domain, thus generalizing knowledge across tasks.} }
Endnote
%0 Conference Paper %T Hierarchical Representation Learning for Markov Decision Processes %A Lorenzo Steccanella %A Simone Totaro %A Anders Jonsson %B Proceedings of The 2nd Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2023 %E Sarath Chandar %E Razvan Pascanu %E Hanie Sedghi %E Doina Precup %F pmlr-v232-steccanella23a %I PMLR %P 568--585 %U https://proceedings.mlr.press/v232/steccanella23a.html %V 232 %X In this paper, we present a novel method for learning reward-agnostic hierarchical representations of Markov Decision Processes. Our method works by partitioning the state space into subsets, and defines subtasks for performing transitions between the partitions. At the high level, we use model-based planning to decide which subtask to pursue next from a given partition. We formulate the problem of partitioning the state space as an optimization problem that can be solved using gradient descent given a set of sampled trajectories, making our method suitable for high-dimensional problems with large state spaces. We empirically validate the method, by showing that it can successfully learn useful hierarchical representations in domains with high-dimensional states. Once learned, the hierarchical representation can be used to solve different tasks in the given domain, thus generalizing knowledge across tasks.
APA
Steccanella, L., Totaro, S. & Jonsson, A.. (2023). Hierarchical Representation Learning for Markov Decision Processes. Proceedings of The 2nd Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 232:568-585 Available from https://proceedings.mlr.press/v232/steccanella23a.html.

Related Material