Neural Program Synthesis from Diverse Demonstration Videos


Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, Joseph Lim ;
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:4790-4799, 2018.


Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans. To empower machines with this ability, we propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos. We introduce a summarizer module as part of our model to improve the network’s ability to integrate multiple demonstrations varying in behavior. We also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training. We show that our model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations. The code is available at

Related Material