PcLast: Discovering Plannable Continuous Latent States

Anurag Koul, Shivakanth Sujit, Shaoru Chen, Ben Evans, Lili Wu, Byron Xu, Rajan Chari, Riashat Islam, Raihan Seraj, Yonathan Efroni, Lekan P Molu, Miroslav Dudı́k, John Langford, Alex Lamb
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:25475-25493, 2024.

Abstract

Goal-conditioned planning benefits from learned low-dimensional representations of rich observations. While compact latent representations typically learned from variational autoencoders or inverse dynamics enable goal-conditioned decision making, they ignore state reachability, hampering their performance. In this paper, we learn a representation that associates reachable states together for effective planning and goal-conditioned policy learning. We first learn a latent representation with multi-step inverse dynamics (to remove distracting information), and then transform this representation to associate reachable states together in ℓ2 space. Our proposals are rigorously tested in various simulation testbeds. Numerical results in reward-based settings show significant improvements in sampling efficiency. Further, in reward-free settings this approach yields layered state abstractions that enable computationally efficient hierarchical planning for reaching ad hoc goals with zero additional samples.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-koul24a, title = {{P}c{L}ast: Discovering Plannable Continuous Latent States}, author = {Koul, Anurag and Sujit, Shivakanth and Chen, Shaoru and Evans, Ben and Wu, Lili and Xu, Byron and Chari, Rajan and Islam, Riashat and Seraj, Raihan and Efroni, Yonathan and Molu, Lekan P and Dud\'{\i}k, Miroslav and Langford, John and Lamb, Alex}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {25475--25493}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/koul24a/koul24a.pdf}, url = {https://proceedings.mlr.press/v235/koul24a.html}, abstract = {Goal-conditioned planning benefits from learned low-dimensional representations of rich observations. While compact latent representations typically learned from variational autoencoders or inverse dynamics enable goal-conditioned decision making, they ignore state reachability, hampering their performance. In this paper, we learn a representation that associates reachable states together for effective planning and goal-conditioned policy learning. We first learn a latent representation with multi-step inverse dynamics (to remove distracting information), and then transform this representation to associate reachable states together in $\ell_2$ space. Our proposals are rigorously tested in various simulation testbeds. Numerical results in reward-based settings show significant improvements in sampling efficiency. Further, in reward-free settings this approach yields layered state abstractions that enable computationally efficient hierarchical planning for reaching ad hoc goals with zero additional samples.} }
Endnote
%0 Conference Paper %T PcLast: Discovering Plannable Continuous Latent States %A Anurag Koul %A Shivakanth Sujit %A Shaoru Chen %A Ben Evans %A Lili Wu %A Byron Xu %A Rajan Chari %A Riashat Islam %A Raihan Seraj %A Yonathan Efroni %A Lekan P Molu %A Miroslav Dudı́k %A John Langford %A Alex Lamb %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-koul24a %I PMLR %P 25475--25493 %U https://proceedings.mlr.press/v235/koul24a.html %V 235 %X Goal-conditioned planning benefits from learned low-dimensional representations of rich observations. While compact latent representations typically learned from variational autoencoders or inverse dynamics enable goal-conditioned decision making, they ignore state reachability, hampering their performance. In this paper, we learn a representation that associates reachable states together for effective planning and goal-conditioned policy learning. We first learn a latent representation with multi-step inverse dynamics (to remove distracting information), and then transform this representation to associate reachable states together in $\ell_2$ space. Our proposals are rigorously tested in various simulation testbeds. Numerical results in reward-based settings show significant improvements in sampling efficiency. Further, in reward-free settings this approach yields layered state abstractions that enable computationally efficient hierarchical planning for reaching ad hoc goals with zero additional samples.
APA
Koul, A., Sujit, S., Chen, S., Evans, B., Wu, L., Xu, B., Chari, R., Islam, R., Seraj, R., Efroni, Y., Molu, L.P., Dudı́k, M., Langford, J. & Lamb, A.. (2024). PcLast: Discovering Plannable Continuous Latent States. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:25475-25493 Available from https://proceedings.mlr.press/v235/koul24a.html.

Related Material