Vector Quantized Models for Planning

Sherjil Ozair, Yazhe Li, Ali Razavi, Ioannis Antonoglou, Aaron Van Den Oord, Oriol Vinyals
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8302-8313, 2021.

Abstract

Recent developments in the field of model-based RL have proven successful in a range of environments, especially ones where planning is essential. However, such successes have been limited to deterministic fully-observed environments. We present a new approach that handles stochastic and partially-observable environments. Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment. We use a stochastic variant of Monte Carlo tree search to plan over both the agent’s actions and the discrete latent variables representing the environment’s response. Our approach significantly outperforms an offline version of MuZero on a stochastic interpretation of chess where the opponent is considered part of the environment. We also show that our approach scales to DeepMind Lab, a first-person 3D environment with large visual observations and partial observability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-ozair21a, title = {Vector Quantized Models for Planning}, author = {Ozair, Sherjil and Li, Yazhe and Razavi, Ali and Antonoglou, Ioannis and Van Den Oord, Aaron and Vinyals, Oriol}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {8302--8313}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/ozair21a/ozair21a.pdf}, url = {https://proceedings.mlr.press/v139/ozair21a.html}, abstract = {Recent developments in the field of model-based RL have proven successful in a range of environments, especially ones where planning is essential. However, such successes have been limited to deterministic fully-observed environments. We present a new approach that handles stochastic and partially-observable environments. Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment. We use a stochastic variant of Monte Carlo tree search to plan over both the agent’s actions and the discrete latent variables representing the environment’s response. Our approach significantly outperforms an offline version of MuZero on a stochastic interpretation of chess where the opponent is considered part of the environment. We also show that our approach scales to DeepMind Lab, a first-person 3D environment with large visual observations and partial observability.} }
Endnote
%0 Conference Paper %T Vector Quantized Models for Planning %A Sherjil Ozair %A Yazhe Li %A Ali Razavi %A Ioannis Antonoglou %A Aaron Van Den Oord %A Oriol Vinyals %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-ozair21a %I PMLR %P 8302--8313 %U https://proceedings.mlr.press/v139/ozair21a.html %V 139 %X Recent developments in the field of model-based RL have proven successful in a range of environments, especially ones where planning is essential. However, such successes have been limited to deterministic fully-observed environments. We present a new approach that handles stochastic and partially-observable environments. Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment. We use a stochastic variant of Monte Carlo tree search to plan over both the agent’s actions and the discrete latent variables representing the environment’s response. Our approach significantly outperforms an offline version of MuZero on a stochastic interpretation of chess where the opponent is considered part of the environment. We also show that our approach scales to DeepMind Lab, a first-person 3D environment with large visual observations and partial observability.
APA
Ozair, S., Li, Y., Razavi, A., Antonoglou, I., Van Den Oord, A. & Vinyals, O.. (2021). Vector Quantized Models for Planning. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:8302-8313 Available from https://proceedings.mlr.press/v139/ozair21a.html.

Related Material