Learning to Infer Generative Template Programs for Visual Concepts

R. Kenny Jones, Siddhartha Chaudhuri, Daniel Ritchie
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:22465-22490, 2024.

Abstract

People grasp flexible visual concepts from a few examples. We explore a neurosymbolic system that learns how to infer programs that capture visual concepts in a domain-general fashion. We introduce Template Programs: programmatic expressions from a domain-specific language that specify structural and parametric patterns common to an input concept. Our framework supports multiple concept-related tasks, including few-shot generation and co-segmentation through parsing. We develop a learning paradigm that allows us to train networks that infer Template Programs directly from visual datasets that contain concept groupings. We run experiments across multiple visual domains: 2D layouts, Omniglot characters, and 3D shapes. We find that our method outperforms task-specific alternatives, and performs competitively against domain-specific approaches for the limited domains where they exist.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-jones24a, title = {Learning to Infer Generative Template Programs for Visual Concepts}, author = {Jones, R. Kenny and Chaudhuri, Siddhartha and Ritchie, Daniel}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {22465--22490}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/jones24a/jones24a.pdf}, url = {https://proceedings.mlr.press/v235/jones24a.html}, abstract = {People grasp flexible visual concepts from a few examples. We explore a neurosymbolic system that learns how to infer programs that capture visual concepts in a domain-general fashion. We introduce Template Programs: programmatic expressions from a domain-specific language that specify structural and parametric patterns common to an input concept. Our framework supports multiple concept-related tasks, including few-shot generation and co-segmentation through parsing. We develop a learning paradigm that allows us to train networks that infer Template Programs directly from visual datasets that contain concept groupings. We run experiments across multiple visual domains: 2D layouts, Omniglot characters, and 3D shapes. We find that our method outperforms task-specific alternatives, and performs competitively against domain-specific approaches for the limited domains where they exist.} }
Endnote
%0 Conference Paper %T Learning to Infer Generative Template Programs for Visual Concepts %A R. Kenny Jones %A Siddhartha Chaudhuri %A Daniel Ritchie %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-jones24a %I PMLR %P 22465--22490 %U https://proceedings.mlr.press/v235/jones24a.html %V 235 %X People grasp flexible visual concepts from a few examples. We explore a neurosymbolic system that learns how to infer programs that capture visual concepts in a domain-general fashion. We introduce Template Programs: programmatic expressions from a domain-specific language that specify structural and parametric patterns common to an input concept. Our framework supports multiple concept-related tasks, including few-shot generation and co-segmentation through parsing. We develop a learning paradigm that allows us to train networks that infer Template Programs directly from visual datasets that contain concept groupings. We run experiments across multiple visual domains: 2D layouts, Omniglot characters, and 3D shapes. We find that our method outperforms task-specific alternatives, and performs competitively against domain-specific approaches for the limited domains where they exist.
APA
Jones, R.K., Chaudhuri, S. & Ritchie, D.. (2024). Learning to Infer Generative Template Programs for Visual Concepts. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:22465-22490 Available from https://proceedings.mlr.press/v235/jones24a.html.

Related Material