Active World Model Learning with Progress Curiosity

Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, Daniel Yamins
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5306-5315, 2020.

Abstract

World models are self-supervised predictive models of how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we construct a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world agents. We propose an AWML system driven by $\gamma$-Progress: a scalable and effective learning progress-based curiosity signal and show that $\gamma$-Progress naturally gives rise to an exploration policy that directs attention to complex but learnable dynamics in a balanced manner, as a result overcoming the “white noise problem”. As a result, our $\gamma$-Progress-driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-kim20e, title = {Active World Model Learning with Progress Curiosity}, author = {Kim, Kuno and Sano, Megumi and De Freitas, Julian and Haber, Nick and Yamins, Daniel}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5306--5315}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/kim20e/kim20e.pdf}, url = {https://proceedings.mlr.press/v119/kim20e.html}, abstract = {World models are self-supervised predictive models of how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we construct a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world agents. We propose an AWML system driven by $\gamma$-Progress: a scalable and effective learning progress-based curiosity signal and show that $\gamma$-Progress naturally gives rise to an exploration policy that directs attention to complex but learnable dynamics in a balanced manner, as a result overcoming the “white noise problem”. As a result, our $\gamma$-Progress-driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.} }
Endnote
%0 Conference Paper %T Active World Model Learning with Progress Curiosity %A Kuno Kim %A Megumi Sano %A Julian De Freitas %A Nick Haber %A Daniel Yamins %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-kim20e %I PMLR %P 5306--5315 %U https://proceedings.mlr.press/v119/kim20e.html %V 119 %X World models are self-supervised predictive models of how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we construct a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world agents. We propose an AWML system driven by $\gamma$-Progress: a scalable and effective learning progress-based curiosity signal and show that $\gamma$-Progress naturally gives rise to an exploration policy that directs attention to complex but learnable dynamics in a balanced manner, as a result overcoming the “white noise problem”. As a result, our $\gamma$-Progress-driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.
APA
Kim, K., Sano, M., De Freitas, J., Haber, N. & Yamins, D.. (2020). Active World Model Learning with Progress Curiosity. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5306-5315 Available from https://proceedings.mlr.press/v119/kim20e.html.

Related Material