Convex and Bilevel Optimization for Neural-Symbolic Inference and Learning

Charles Andrew Dickens, Changyu Gao, Connor Pryor, Stephen Wright, Lise Getoor
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:10865-10896, 2024.

Abstract

We leverage convex and bilevel optimization techniques to develop a general gradient-based parameter learning framework for neural-symbolic (NeSy) systems. We demonstrate our framework with NeuPSL, a state-of-the-art NeSy architecture. To achieve this, we propose a smooth primal and dual formulation of NeuPSL inference and show learning gradients are functions of the optimal dual variables. Additionally, we develop a dual block coordinate descent algorithm for the new formulation that naturally exploits warm-starts. This leads to over $100 \times$ learning runtime improvements over the current best NeuPSL inference method. Finally, we provide extensive empirical evaluations across $8$ datasets covering a range of tasks and demonstrate our learning framework achieves up to a $16$% point prediction performance improvement over alternative learning methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-dickens24a, title = {Convex and Bilevel Optimization for Neural-Symbolic Inference and Learning}, author = {Dickens, Charles Andrew and Gao, Changyu and Pryor, Connor and Wright, Stephen and Getoor, Lise}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {10865--10896}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/dickens24a/dickens24a.pdf}, url = {https://proceedings.mlr.press/v235/dickens24a.html}, abstract = {We leverage convex and bilevel optimization techniques to develop a general gradient-based parameter learning framework for neural-symbolic (NeSy) systems. We demonstrate our framework with NeuPSL, a state-of-the-art NeSy architecture. To achieve this, we propose a smooth primal and dual formulation of NeuPSL inference and show learning gradients are functions of the optimal dual variables. Additionally, we develop a dual block coordinate descent algorithm for the new formulation that naturally exploits warm-starts. This leads to over $100 \times$ learning runtime improvements over the current best NeuPSL inference method. Finally, we provide extensive empirical evaluations across $8$ datasets covering a range of tasks and demonstrate our learning framework achieves up to a $16$% point prediction performance improvement over alternative learning methods.} }
Endnote
%0 Conference Paper %T Convex and Bilevel Optimization for Neural-Symbolic Inference and Learning %A Charles Andrew Dickens %A Changyu Gao %A Connor Pryor %A Stephen Wright %A Lise Getoor %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-dickens24a %I PMLR %P 10865--10896 %U https://proceedings.mlr.press/v235/dickens24a.html %V 235 %X We leverage convex and bilevel optimization techniques to develop a general gradient-based parameter learning framework for neural-symbolic (NeSy) systems. We demonstrate our framework with NeuPSL, a state-of-the-art NeSy architecture. To achieve this, we propose a smooth primal and dual formulation of NeuPSL inference and show learning gradients are functions of the optimal dual variables. Additionally, we develop a dual block coordinate descent algorithm for the new formulation that naturally exploits warm-starts. This leads to over $100 \times$ learning runtime improvements over the current best NeuPSL inference method. Finally, we provide extensive empirical evaluations across $8$ datasets covering a range of tasks and demonstrate our learning framework achieves up to a $16$% point prediction performance improvement over alternative learning methods.
APA
Dickens, C.A., Gao, C., Pryor, C., Wright, S. & Getoor, L.. (2024). Convex and Bilevel Optimization for Neural-Symbolic Inference and Learning. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:10865-10896 Available from https://proceedings.mlr.press/v235/dickens24a.html.

Related Material