Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation

Sebastian Brechtel, Tobias Gindele, Rüdiger Dillmann
Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):370-378, 2013.

Abstract

Discrete POMDPs of medium complexity can be approximately solved in reasonable time. However, most applications have a continuous and thus uncountably infinite state space. We propose the novel concept of learning a discrete representation of the continuous state space to solve the integrals in continuous POMDPs efficiently and generalize sparse calculations over the continuous space. The representation is iteratively refined as part of a novel Value Iteration step and does not depend on prior knowledge. Consistency for the learned generalization is asserted by a self-correction algorithm. The presented concept is implemented for continuous state and observation spaces based on Monte Carlo approximation to allow for arbitrary POMDP models. In an experimental comparison it yields higher values in significantly shorter time than state of the art algorithms and solves higher-dimensional problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v28-brechtel13, title = {Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation}, author = {Brechtel, Sebastian and Gindele, Tobias and Dillmann, Rüdiger}, booktitle = {Proceedings of the 30th International Conference on Machine Learning}, pages = {370--378}, year = {2013}, editor = {Dasgupta, Sanjoy and McAllester, David}, volume = {28}, number = {3}, series = {Proceedings of Machine Learning Research}, address = {Atlanta, Georgia, USA}, month = {17--19 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v28/brechtel13.pdf}, url = {https://proceedings.mlr.press/v28/brechtel13.html}, abstract = {Discrete POMDPs of medium complexity can be approximately solved in reasonable time. However, most applications have a continuous and thus uncountably infinite state space. We propose the novel concept of learning a discrete representation of the continuous state space to solve the integrals in continuous POMDPs efficiently and generalize sparse calculations over the continuous space. The representation is iteratively refined as part of a novel Value Iteration step and does not depend on prior knowledge. Consistency for the learned generalization is asserted by a self-correction algorithm. The presented concept is implemented for continuous state and observation spaces based on Monte Carlo approximation to allow for arbitrary POMDP models. In an experimental comparison it yields higher values in significantly shorter time than state of the art algorithms and solves higher-dimensional problems.} }
Endnote
%0 Conference Paper %T Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation %A Sebastian Brechtel %A Tobias Gindele %A Rüdiger Dillmann %B Proceedings of the 30th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2013 %E Sanjoy Dasgupta %E David McAllester %F pmlr-v28-brechtel13 %I PMLR %P 370--378 %U https://proceedings.mlr.press/v28/brechtel13.html %V 28 %N 3 %X Discrete POMDPs of medium complexity can be approximately solved in reasonable time. However, most applications have a continuous and thus uncountably infinite state space. We propose the novel concept of learning a discrete representation of the continuous state space to solve the integrals in continuous POMDPs efficiently and generalize sparse calculations over the continuous space. The representation is iteratively refined as part of a novel Value Iteration step and does not depend on prior knowledge. Consistency for the learned generalization is asserted by a self-correction algorithm. The presented concept is implemented for continuous state and observation spaces based on Monte Carlo approximation to allow for arbitrary POMDP models. In an experimental comparison it yields higher values in significantly shorter time than state of the art algorithms and solves higher-dimensional problems.
RIS
TY - CPAPER TI - Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation AU - Sebastian Brechtel AU - Tobias Gindele AU - Rüdiger Dillmann BT - Proceedings of the 30th International Conference on Machine Learning DA - 2013/05/26 ED - Sanjoy Dasgupta ED - David McAllester ID - pmlr-v28-brechtel13 PB - PMLR DP - Proceedings of Machine Learning Research VL - 28 IS - 3 SP - 370 EP - 378 L1 - http://proceedings.mlr.press/v28/brechtel13.pdf UR - https://proceedings.mlr.press/v28/brechtel13.html AB - Discrete POMDPs of medium complexity can be approximately solved in reasonable time. However, most applications have a continuous and thus uncountably infinite state space. We propose the novel concept of learning a discrete representation of the continuous state space to solve the integrals in continuous POMDPs efficiently and generalize sparse calculations over the continuous space. The representation is iteratively refined as part of a novel Value Iteration step and does not depend on prior knowledge. Consistency for the learned generalization is asserted by a self-correction algorithm. The presented concept is implemented for continuous state and observation spaces based on Monte Carlo approximation to allow for arbitrary POMDP models. In an experimental comparison it yields higher values in significantly shorter time than state of the art algorithms and solves higher-dimensional problems. ER -
APA
Brechtel, S., Gindele, T. & Dillmann, R.. (2013). Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation. Proceedings of the 30th International Conference on Machine Learning, in Proceedings of Machine Learning Research 28(3):370-378 Available from https://proceedings.mlr.press/v28/brechtel13.html.

Related Material