The Predictron: End-To-End Learning and Planning

David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, Thomas Degris
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3191-3199, 2017.

Abstract

One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple “imagined” planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-silver17a, title = {The Predictron: End-To-End Learning and Planning}, author = {David Silver and Hado van Hasselt and Matteo Hessel and Tom Schaul and Arthur Guez and Tim Harley and Gabriel Dulac-Arnold and David Reichert and Neil Rabinowitz and Andre Barreto and Thomas Degris}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3191--3199}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/silver17a/silver17a.pdf}, url = {https://proceedings.mlr.press/v70/silver17a.html}, abstract = {One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple “imagined” planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.} }
Endnote
%0 Conference Paper %T The Predictron: End-To-End Learning and Planning %A David Silver %A Hado Hasselt %A Matteo Hessel %A Tom Schaul %A Arthur Guez %A Tim Harley %A Gabriel Dulac-Arnold %A David Reichert %A Neil Rabinowitz %A Andre Barreto %A Thomas Degris %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-silver17a %I PMLR %P 3191--3199 %U https://proceedings.mlr.press/v70/silver17a.html %V 70 %X One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple “imagined” planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.
APA
Silver, D., Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N., Barreto, A. & Degris, T.. (2017). The Predictron: End-To-End Learning and Planning. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3191-3199 Available from https://proceedings.mlr.press/v70/silver17a.html.

Related Material