Generating Programmatic Referring Expressions via Program Synthesis

Jiani Huang, Calvin Smith, Osbert Bastani, Rishabh Singh, Aws Albarghouthi, Mayur Naik
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4495-4506, 2020.

Abstract

Incorporating symbolic reasoning into machine learning algorithms is a promising approach to improve performance on learning tasks that require logical reasoning. We study the problem of generating a programmatic variant of referring expressions that we call referring relational programs. In particular, given a symbolic representation of an image and a target object in that image, the goal is to generate a relational program that uniquely identifies the target object in terms of its attributes and its relations to other objects in the image. We propose a neurosymbolic program synthesis algorithm that combines a policy neural network with enumerative search to generate such relational programs. The policy neural network employs a program interpreter that provides immediate feedback on the consequences of the decisions made by the policy, and also takes into account the uncertainty in the symbolic representation of the image. We evaluate our algorithm on challenging benchmarks based on the CLEVR dataset, and demonstrate that our approach significantly outperforms several baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-huang20h, title = {Generating Programmatic Referring Expressions via Program Synthesis}, author = {Huang, Jiani and Smith, Calvin and Bastani, Osbert and Singh, Rishabh and Albarghouthi, Aws and Naik, Mayur}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {4495--4506}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/huang20h/huang20h.pdf}, url = {https://proceedings.mlr.press/v119/huang20h.html}, abstract = {Incorporating symbolic reasoning into machine learning algorithms is a promising approach to improve performance on learning tasks that require logical reasoning. We study the problem of generating a programmatic variant of referring expressions that we call referring relational programs. In particular, given a symbolic representation of an image and a target object in that image, the goal is to generate a relational program that uniquely identifies the target object in terms of its attributes and its relations to other objects in the image. We propose a neurosymbolic program synthesis algorithm that combines a policy neural network with enumerative search to generate such relational programs. The policy neural network employs a program interpreter that provides immediate feedback on the consequences of the decisions made by the policy, and also takes into account the uncertainty in the symbolic representation of the image. We evaluate our algorithm on challenging benchmarks based on the CLEVR dataset, and demonstrate that our approach significantly outperforms several baselines.} }
Endnote
%0 Conference Paper %T Generating Programmatic Referring Expressions via Program Synthesis %A Jiani Huang %A Calvin Smith %A Osbert Bastani %A Rishabh Singh %A Aws Albarghouthi %A Mayur Naik %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-huang20h %I PMLR %P 4495--4506 %U https://proceedings.mlr.press/v119/huang20h.html %V 119 %X Incorporating symbolic reasoning into machine learning algorithms is a promising approach to improve performance on learning tasks that require logical reasoning. We study the problem of generating a programmatic variant of referring expressions that we call referring relational programs. In particular, given a symbolic representation of an image and a target object in that image, the goal is to generate a relational program that uniquely identifies the target object in terms of its attributes and its relations to other objects in the image. We propose a neurosymbolic program synthesis algorithm that combines a policy neural network with enumerative search to generate such relational programs. The policy neural network employs a program interpreter that provides immediate feedback on the consequences of the decisions made by the policy, and also takes into account the uncertainty in the symbolic representation of the image. We evaluate our algorithm on challenging benchmarks based on the CLEVR dataset, and demonstrate that our approach significantly outperforms several baselines.
APA
Huang, J., Smith, C., Bastani, O., Singh, R., Albarghouthi, A. & Naik, M.. (2020). Generating Programmatic Referring Expressions via Program Synthesis. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:4495-4506 Available from https://proceedings.mlr.press/v119/huang20h.html.

Related Material