MDPs with Unawareness in Robotics

Nan Rong Cornell University, Joseph Halpern Cornell University, Ashutosh Saxena Cornell University
Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, PMLR R14:811-820, 2016.

Abstract

We formalize decision-making problems in robotics and automated control using continuous MDPs and actions that take place over continuous time intervals. We then approximate the continuous MDP using finer and finer discretizations. Doing this results in a family of systems, each of which has an extremely large action space, although only a few actions are "interesting". We can view the decision maker as being unaware of which actions are "interesting". We an model this using MDPUs, MDPs with unawareness, where the action space is much smaller. As we show, MDPUs can be used as a general framework for learning tasks in robotic problems. We prove results on the difficulty of learning a near-optimal policy in an an MDPU for a continuous task. We apply these ideas to the problem of having a humanoid robot learn on its own how to walk.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR14-university16q, title = {MDPs with Unawareness in Robotics}, author = {University, Nan Rong Cornell and University, Joseph Halpern Cornell and University, Ashutosh Saxena Cornell}, booktitle = {Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence}, pages = {811--820}, year = {2016}, editor = {Ihler, Alexander and Janzing, Dominik}, volume = {R14}, series = {Proceedings of Machine Learning Research}, month = {25--29 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r14/main/assets/university16q/university16q.pdf}, url = {https://proceedings.mlr.press/r14/university16q.html}, abstract = {We formalize decision-making problems in robotics and automated control using continuous MDPs and actions that take place over continuous time intervals. We then approximate the continuous MDP using finer and finer discretizations. Doing this results in a family of systems, each of which has an extremely large action space, although only a few actions are "interesting". We can view the decision maker as being unaware of which actions are "interesting". We an model this using MDPUs, MDPs with unawareness, where the action space is much smaller. As we show, MDPUs can be used as a general framework for learning tasks in robotic problems. We prove results on the difficulty of learning a near-optimal policy in an an MDPU for a continuous task. We apply these ideas to the problem of having a humanoid robot learn on its own how to walk.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T MDPs with Unawareness in Robotics %A Nan Rong Cornell University %A Joseph Halpern Cornell University %A Ashutosh Saxena Cornell University %B Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2016 %E Alexander Ihler %E Dominik Janzing %F pmlr-vR14-university16q %I PMLR %P 811--820 %U https://proceedings.mlr.press/r14/university16q.html %V R14 %X We formalize decision-making problems in robotics and automated control using continuous MDPs and actions that take place over continuous time intervals. We then approximate the continuous MDP using finer and finer discretizations. Doing this results in a family of systems, each of which has an extremely large action space, although only a few actions are "interesting". We can view the decision maker as being unaware of which actions are "interesting". We an model this using MDPUs, MDPs with unawareness, where the action space is much smaller. As we show, MDPUs can be used as a general framework for learning tasks in robotic problems. We prove results on the difficulty of learning a near-optimal policy in an an MDPU for a continuous task. We apply these ideas to the problem of having a humanoid robot learn on its own how to walk. %Z Reissued by PMLR on 04 October 2026.
APA
University, N.R.C., University, J.H.C. & University, A.S.C.. (2016). MDPs with Unawareness in Robotics. Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R14:811-820 Available from https://proceedings.mlr.press/r14/university16q.html. Reissued by PMLR on 04 October 2026.

Related Material