Learning Compact Models for Planning with Exogenous Processes

Rohan Chitnis, Tomás Lozano-Pérez
Proceedings of the Conference on Robot Learning, PMLR 100:813-822, 2020.

Abstract

We address the problem of approximate model minimization for MDPs in which the state is partitioned into endogenous and (much larger) exogenous components. An exogenous state variable is one whose dynamics are independent of the agent’s actions. We formalize the mask-learning problem, in which the agent must choose a subset of exogenous state variables to reason about when planning; doing planning in such a reduced state space can often be significantly more efficient than planning in the full model. We then explore the various value functions at play within this setting, and describe conditions under which a policy for a reduced model will be optimal for the full MDP. The analysis leads us to a tractable approximate algorithm that draws upon the notion of mutual information among exogenous state variables. We validate our approach in simulated robotic manipulation domains where a robot is placed in a busy environment, in which there are many other agents also interacting with the objects. Visit http://tinyurl.com/chitnis-exogenous for a supplementary video.

Cite this Paper

BibTeX
@InProceedings{pmlr-v100-chitnis20a, title = {Learning Compact Models for Planning with Exogenous Processes}, author = {Chitnis, Rohan and Lozano-P\'erez, Tom\'as}, booktitle = {Proceedings of the Conference on Robot Learning}, pages = {813--822}, year = {2020}, editor = {Kaelbling, Leslie Pack and Kragic, Danica and Sugiura, Komei}, volume = {100}, series = {Proceedings of Machine Learning Research}, month = {30 Oct--01 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v100/chitnis20a/chitnis20a.pdf}, url = {https://proceedings.mlr.press/v100/chitnis20a.html}, abstract = {We address the problem of approximate model minimization for MDPs in which the state is partitioned into endogenous and (much larger) exogenous components. An exogenous state variable is one whose dynamics are independent of the agent’s actions. We formalize the mask-learning problem, in which the agent must choose a subset of exogenous state variables to reason about when planning; doing planning in such a reduced state space can often be significantly more efficient than planning in the full model. We then explore the various value functions at play within this setting, and describe conditions under which a policy for a reduced model will be optimal for the full MDP. The analysis leads us to a tractable approximate algorithm that draws upon the notion of mutual information among exogenous state variables. We validate our approach in simulated robotic manipulation domains where a robot is placed in a busy environment, in which there are many other agents also interacting with the objects. Visit http://tinyurl.com/chitnis-exogenous for a supplementary video.} }
Endnote
%0 Conference Paper %T Learning Compact Models for Planning with Exogenous Processes %A Rohan Chitnis %A Tomás Lozano-Pérez %B Proceedings of the Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2020 %E Leslie Pack Kaelbling %E Danica Kragic %E Komei Sugiura %F pmlr-v100-chitnis20a %I PMLR %P 813--822 %U https://proceedings.mlr.press/v100/chitnis20a.html %V 100 %X We address the problem of approximate model minimization for MDPs in which the state is partitioned into endogenous and (much larger) exogenous components. An exogenous state variable is one whose dynamics are independent of the agent’s actions. We formalize the mask-learning problem, in which the agent must choose a subset of exogenous state variables to reason about when planning; doing planning in such a reduced state space can often be significantly more efficient than planning in the full model. We then explore the various value functions at play within this setting, and describe conditions under which a policy for a reduced model will be optimal for the full MDP. The analysis leads us to a tractable approximate algorithm that draws upon the notion of mutual information among exogenous state variables. We validate our approach in simulated robotic manipulation domains where a robot is placed in a busy environment, in which there are many other agents also interacting with the objects. Visit http://tinyurl.com/chitnis-exogenous for a supplementary video.
APA
Chitnis, R. & Lozano-Pérez, T.. (2020). Learning Compact Models for Planning with Exogenous Processes. Proceedings of the Conference on Robot Learning, in Proceedings of Machine Learning Research 100:813-822 Available from https://proceedings.mlr.press/v100/chitnis20a.html.

Related Material