Model-Based Relational RL When Object Existence is Partially Observable

Ngo Ahn Vien, Marc Toussaint
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):559-567, 2014.

Abstract

We consider learning and planning in relational MDPs when object existence is uncertain and new objects may appear or disappear depending on previous actions or properties of other objects. Optimal policies actively need to discover objects to achieve a goal; planning in such domains in general amounts to a POMDP problem, where the belief is about the existence and properties of potential not-yet-discovered objects. We propose a computationally efficient extension of model-based relational RL methods that approximates these beliefs using discrete uncertainty predicates. In this formulation the belief update is learned using probabilistic rules and planning in the approximated belief space can be achieved using an extension of existing planners. We prove that the learned belief update rules encode an approximation of the exact belief updates of a POMDP formulation and demonstrate experimentally that the proposed approach successfully learns a set of relational rules appropriate to solve such problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-ngo14, title = {Model-Based Relational RL When Object Existence is Partially Observable}, author = {Vien, Ngo Ahn and Toussaint, Marc}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {559--567}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/ngo14.pdf}, url = {https://proceedings.mlr.press/v32/ngo14.html}, abstract = {We consider learning and planning in relational MDPs when object existence is uncertain and new objects may appear or disappear depending on previous actions or properties of other objects. Optimal policies actively need to discover objects to achieve a goal; planning in such domains in general amounts to a POMDP problem, where the belief is about the existence and properties of potential not-yet-discovered objects. We propose a computationally efficient extension of model-based relational RL methods that approximates these beliefs using discrete uncertainty predicates. In this formulation the belief update is learned using probabilistic rules and planning in the approximated belief space can be achieved using an extension of existing planners. We prove that the learned belief update rules encode an approximation of the exact belief updates of a POMDP formulation and demonstrate experimentally that the proposed approach successfully learns a set of relational rules appropriate to solve such problems.} }
Endnote
%0 Conference Paper %T Model-Based Relational RL When Object Existence is Partially Observable %A Ngo Ahn Vien %A Marc Toussaint %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-ngo14 %I PMLR %P 559--567 %U https://proceedings.mlr.press/v32/ngo14.html %V 32 %N 2 %X We consider learning and planning in relational MDPs when object existence is uncertain and new objects may appear or disappear depending on previous actions or properties of other objects. Optimal policies actively need to discover objects to achieve a goal; planning in such domains in general amounts to a POMDP problem, where the belief is about the existence and properties of potential not-yet-discovered objects. We propose a computationally efficient extension of model-based relational RL methods that approximates these beliefs using discrete uncertainty predicates. In this formulation the belief update is learned using probabilistic rules and planning in the approximated belief space can be achieved using an extension of existing planners. We prove that the learned belief update rules encode an approximation of the exact belief updates of a POMDP formulation and demonstrate experimentally that the proposed approach successfully learns a set of relational rules appropriate to solve such problems.
RIS
TY - CPAPER TI - Model-Based Relational RL When Object Existence is Partially Observable AU - Ngo Ahn Vien AU - Marc Toussaint BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-ngo14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 559 EP - 567 L1 - http://proceedings.mlr.press/v32/ngo14.pdf UR - https://proceedings.mlr.press/v32/ngo14.html AB - We consider learning and planning in relational MDPs when object existence is uncertain and new objects may appear or disappear depending on previous actions or properties of other objects. Optimal policies actively need to discover objects to achieve a goal; planning in such domains in general amounts to a POMDP problem, where the belief is about the existence and properties of potential not-yet-discovered objects. We propose a computationally efficient extension of model-based relational RL methods that approximates these beliefs using discrete uncertainty predicates. In this formulation the belief update is learned using probabilistic rules and planning in the approximated belief space can be achieved using an extension of existing planners. We prove that the learned belief update rules encode an approximation of the exact belief updates of a POMDP formulation and demonstrate experimentally that the proposed approach successfully learns a set of relational rules appropriate to solve such problems. ER -
APA
Vien, N.A. & Toussaint, M.. (2014). Model-Based Relational RL When Object Existence is Partially Observable. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):559-567 Available from https://proceedings.mlr.press/v32/ngo14.html.

Related Material