Bayesian Reinforcement Learning for Single-Episode Missions in Partially Unknown Environments

Matthew Budd, Paul Duckworth, Nick Hawes, Bruno Lacerda
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1189-1198, 2023.

Abstract

We consider planning for mobile robots conducting missions in real-world domains where a priori unknown dynamics affect the robot’s costs and transitions. We study single-episode missions where it is crucial that the robot appropriately trades off exploration and exploitation, such that the learning of the environment dynamics is just enough to effectively complete the mission. Thus, we propose modelling unknown dynamics using Gaussian processes, which provide a principled Bayesian framework for incorporating online observations made by the robot, and using them to predict the dynamics in unexplored areas. We then formulate the problem of mission planning in Markov decision processes under Gaussian process predictions as Bayesian model-based reinforcement learning. This allows us to employ solution techniques that plan more efficiently than previous Gaussian process planning methods are able to. We empirically evaluate the benefits of our formulation in an underwater autonomous vehicle navigation task and robot mission planning in a realistic simulation of a nuclear environment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-budd23a, title = {Bayesian Reinforcement Learning for Single-Episode Missions in Partially Unknown Environments}, author = {Budd, Matthew and Duckworth, Paul and Hawes, Nick and Lacerda, Bruno}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1189--1198}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/budd23a/budd23a.pdf}, url = {https://proceedings.mlr.press/v205/budd23a.html}, abstract = {We consider planning for mobile robots conducting missions in real-world domains where a priori unknown dynamics affect the robot’s costs and transitions. We study single-episode missions where it is crucial that the robot appropriately trades off exploration and exploitation, such that the learning of the environment dynamics is just enough to effectively complete the mission. Thus, we propose modelling unknown dynamics using Gaussian processes, which provide a principled Bayesian framework for incorporating online observations made by the robot, and using them to predict the dynamics in unexplored areas. We then formulate the problem of mission planning in Markov decision processes under Gaussian process predictions as Bayesian model-based reinforcement learning. This allows us to employ solution techniques that plan more efficiently than previous Gaussian process planning methods are able to. We empirically evaluate the benefits of our formulation in an underwater autonomous vehicle navigation task and robot mission planning in a realistic simulation of a nuclear environment.} }
Endnote
%0 Conference Paper %T Bayesian Reinforcement Learning for Single-Episode Missions in Partially Unknown Environments %A Matthew Budd %A Paul Duckworth %A Nick Hawes %A Bruno Lacerda %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-budd23a %I PMLR %P 1189--1198 %U https://proceedings.mlr.press/v205/budd23a.html %V 205 %X We consider planning for mobile robots conducting missions in real-world domains where a priori unknown dynamics affect the robot’s costs and transitions. We study single-episode missions where it is crucial that the robot appropriately trades off exploration and exploitation, such that the learning of the environment dynamics is just enough to effectively complete the mission. Thus, we propose modelling unknown dynamics using Gaussian processes, which provide a principled Bayesian framework for incorporating online observations made by the robot, and using them to predict the dynamics in unexplored areas. We then formulate the problem of mission planning in Markov decision processes under Gaussian process predictions as Bayesian model-based reinforcement learning. This allows us to employ solution techniques that plan more efficiently than previous Gaussian process planning methods are able to. We empirically evaluate the benefits of our formulation in an underwater autonomous vehicle navigation task and robot mission planning in a realistic simulation of a nuclear environment.
APA
Budd, M., Duckworth, P., Hawes, N. & Lacerda, B.. (2023). Bayesian Reinforcement Learning for Single-Episode Missions in Partially Unknown Environments. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1189-1198 Available from https://proceedings.mlr.press/v205/budd23a.html.

Related Material