Curious iLQR: Resolving Uncertainty in Model-based RL

Sarah Bechtle, Yixin Lin, Akshara Rai, Ludovic Righetti, Franziska Meier
Proceedings of the Conference on Robot Learning, PMLR 100:162-171, 2020.

Abstract

Curiosity as a means to explore during reinforcement learning problems has recently become very popular. However, very little progress has been made in utilizing curiosity for learning control. In this work, we propose a model-based reinforcement learning (MBRL) framework that combines Bayesian modeling of the system dynamics with curious iLQR , an iterative LQR approach that considers model uncertainty. During trajectory optimization the curious iLQR attempts to minimize both the task-dependent cost and the uncertainty in the dynamics model. We demonstrate the approach on reaching tasks with 7-DoF manipulators in simulation and on a real robot. Our experiments show that MBRL with curious iLQR reaches desired end-effector targets more reliably and with less system rollouts when learning a new task from scratch, and that the learned model generalizes better to new reaching tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v100-bechtle20a, title = {Curious iLQR: Resolving Uncertainty in Model-based RL}, author = {Bechtle, Sarah and Lin, Yixin and Rai, Akshara and Righetti, Ludovic and Meier, Franziska}, booktitle = {Proceedings of the Conference on Robot Learning}, pages = {162--171}, year = {2020}, editor = {Kaelbling, Leslie Pack and Kragic, Danica and Sugiura, Komei}, volume = {100}, series = {Proceedings of Machine Learning Research}, month = {30 Oct--01 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v100/bechtle20a/bechtle20a.pdf}, url = {https://proceedings.mlr.press/v100/bechtle20a.html}, abstract = {Curiosity as a means to explore during reinforcement learning problems has recently become very popular. However, very little progress has been made in utilizing curiosity for learning control. In this work, we propose a model-based reinforcement learning (MBRL) framework that combines Bayesian modeling of the system dynamics with curious iLQR , an iterative LQR approach that considers model uncertainty. During trajectory optimization the curious iLQR attempts to minimize both the task-dependent cost and the uncertainty in the dynamics model. We demonstrate the approach on reaching tasks with 7-DoF manipulators in simulation and on a real robot. Our experiments show that MBRL with curious iLQR reaches desired end-effector targets more reliably and with less system rollouts when learning a new task from scratch, and that the learned model generalizes better to new reaching tasks.} }
Endnote
%0 Conference Paper %T Curious iLQR: Resolving Uncertainty in Model-based RL %A Sarah Bechtle %A Yixin Lin %A Akshara Rai %A Ludovic Righetti %A Franziska Meier %B Proceedings of the Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2020 %E Leslie Pack Kaelbling %E Danica Kragic %E Komei Sugiura %F pmlr-v100-bechtle20a %I PMLR %P 162--171 %U https://proceedings.mlr.press/v100/bechtle20a.html %V 100 %X Curiosity as a means to explore during reinforcement learning problems has recently become very popular. However, very little progress has been made in utilizing curiosity for learning control. In this work, we propose a model-based reinforcement learning (MBRL) framework that combines Bayesian modeling of the system dynamics with curious iLQR , an iterative LQR approach that considers model uncertainty. During trajectory optimization the curious iLQR attempts to minimize both the task-dependent cost and the uncertainty in the dynamics model. We demonstrate the approach on reaching tasks with 7-DoF manipulators in simulation and on a real robot. Our experiments show that MBRL with curious iLQR reaches desired end-effector targets more reliably and with less system rollouts when learning a new task from scratch, and that the learned model generalizes better to new reaching tasks.
APA
Bechtle, S., Lin, Y., Rai, A., Righetti, L. & Meier, F.. (2020). Curious iLQR: Resolving Uncertainty in Model-based RL. Proceedings of the Conference on Robot Learning, in Proceedings of Machine Learning Research 100:162-171 Available from https://proceedings.mlr.press/v100/bechtle20a.html.

Related Material