Task-Aware Exploration via a Predictive Bisimulation Metric

Dayang Liang, Ruihan Liu, Lipeng Wan, Yunlong Liu, Bo An
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:73336-73355, 2026.

Abstract

Accelerating exploration in visual reinforcement learning under sparse rewards remains challenging due to the substantial task-irrelevant variations. Despite advances in intrinsic exploration, many methods either assume access to low-dimensional states or lack task-aware exploration strategies, thereby rendering them fragile in visual domains. To bridge this gap, we present TEB, a Task-aware Exploration approach that tightly couples task-relevant representations with exploration through a predictive Bisimulation metric. Specifically, TEB leverages the metric not only to learn behaviorally grounded task representations but also to measure global intrinsic novelty over the learned latent space. To realize this, we first theoretically mitigate the representation collapse of degenerate bisimulation metrics under sparse rewards by internally introducing a simple but effective predicted reward differential term. Building on this robust metric, we further introduce potential-based global exploration bonuses over anchor states, which measure the relative novelty between observations over the latent space. Extensive experiments on MetaWorld and Maze2D show that TEB achieves superior exploration ability and outperforms recent baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-liang26o, title = {Task-Aware Exploration via a Predictive Bisimulation Metric}, author = {Liang, Dayang and Liu, Ruihan and Wan, Lipeng and Liu, Yunlong and An, Bo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {73336--73355}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liang26o/liang26o.pdf}, url = {https://proceedings.mlr.press/v306/liang26o.html}, abstract = {Accelerating exploration in visual reinforcement learning under sparse rewards remains challenging due to the substantial task-irrelevant variations. Despite advances in intrinsic exploration, many methods either assume access to low-dimensional states or lack task-aware exploration strategies, thereby rendering them fragile in visual domains. To bridge this gap, we present TEB, a Task-aware Exploration approach that tightly couples task-relevant representations with exploration through a predictive Bisimulation metric. Specifically, TEB leverages the metric not only to learn behaviorally grounded task representations but also to measure global intrinsic novelty over the learned latent space. To realize this, we first theoretically mitigate the representation collapse of degenerate bisimulation metrics under sparse rewards by internally introducing a simple but effective predicted reward differential term. Building on this robust metric, we further introduce potential-based global exploration bonuses over anchor states, which measure the relative novelty between observations over the latent space. Extensive experiments on MetaWorld and Maze2D show that TEB achieves superior exploration ability and outperforms recent baselines.} }
Endnote
%0 Conference Paper %T Task-Aware Exploration via a Predictive Bisimulation Metric %A Dayang Liang %A Ruihan Liu %A Lipeng Wan %A Yunlong Liu %A Bo An %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liang26o %I PMLR %P 73336--73355 %U https://proceedings.mlr.press/v306/liang26o.html %V 306 %X Accelerating exploration in visual reinforcement learning under sparse rewards remains challenging due to the substantial task-irrelevant variations. Despite advances in intrinsic exploration, many methods either assume access to low-dimensional states or lack task-aware exploration strategies, thereby rendering them fragile in visual domains. To bridge this gap, we present TEB, a Task-aware Exploration approach that tightly couples task-relevant representations with exploration through a predictive Bisimulation metric. Specifically, TEB leverages the metric not only to learn behaviorally grounded task representations but also to measure global intrinsic novelty over the learned latent space. To realize this, we first theoretically mitigate the representation collapse of degenerate bisimulation metrics under sparse rewards by internally introducing a simple but effective predicted reward differential term. Building on this robust metric, we further introduce potential-based global exploration bonuses over anchor states, which measure the relative novelty between observations over the latent space. Extensive experiments on MetaWorld and Maze2D show that TEB achieves superior exploration ability and outperforms recent baselines.
APA
Liang, D., Liu, R., Wan, L., Liu, Y. & An, B.. (2026). Task-Aware Exploration via a Predictive Bisimulation Metric. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:73336-73355 Available from https://proceedings.mlr.press/v306/liang26o.html.

Related Material