Hierarchical Reinforcement Learning with Targeted Causal Interventions

Mohammadsadegh Khorasani, Saber Salehkaleybar, Negar Kiyavash, Matthias Grossglauser
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:29869-29912, 2025.

Abstract

Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erdős-Rényi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-khorasani25a, title = {Hierarchical Reinforcement Learning with Targeted Causal Interventions}, author = {Khorasani, Mohammadsadegh and Salehkaleybar, Saber and Kiyavash, Negar and Grossglauser, Matthias}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {29869--29912}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/khorasani25a/khorasani25a.pdf}, url = {https://proceedings.mlr.press/v267/khorasani25a.html}, abstract = {Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erdős-Rényi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.} }
Endnote
%0 Conference Paper %T Hierarchical Reinforcement Learning with Targeted Causal Interventions %A Mohammadsadegh Khorasani %A Saber Salehkaleybar %A Negar Kiyavash %A Matthias Grossglauser %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-khorasani25a %I PMLR %P 29869--29912 %U https://proceedings.mlr.press/v267/khorasani25a.html %V 267 %X Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erdős-Rényi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.
APA
Khorasani, M., Salehkaleybar, S., Kiyavash, N. & Grossglauser, M.. (2025). Hierarchical Reinforcement Learning with Targeted Causal Interventions. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:29869-29912 Available from https://proceedings.mlr.press/v267/khorasani25a.html.

Related Material