Injecting Logical Constraints into Neural Networks via Straight-Through Estimators

Zhun Yang, Joohyung Lee, Chiyoun Park
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:25096-25122, 2022.

Abstract

Injecting discrete logical constraints into neural network learning is one of the main challenges in neuro-symbolic AI. We find that a straight-through-estimator, a method introduced to train binary neural networks, could effectively be applied to incorporate logical constraints into neural network learning. More specifically, we design a systematic way to represent discrete logical constraints as a loss function; minimizing this loss using gradient descent via a straight-through-estimator updates the neural network’s weights in the direction that the binarized outputs satisfy the logical constraints. The experimental results show that by leveraging GPUs and batch training, this method scales significantly better than existing neuro-symbolic methods that require heavy symbolic computation for computing gradients. Also, we demonstrate that our method applies to different types of neural networks, such as MLP, CNN, and GNN, making them learn with no or fewer labeled data by learning directly from known constraints.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-yang22h, title = {Injecting Logical Constraints into Neural Networks via Straight-Through Estimators}, author = {Yang, Zhun and Lee, Joohyung and Park, Chiyoun}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {25096--25122}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/yang22h/yang22h.pdf}, url = {https://proceedings.mlr.press/v162/yang22h.html}, abstract = {Injecting discrete logical constraints into neural network learning is one of the main challenges in neuro-symbolic AI. We find that a straight-through-estimator, a method introduced to train binary neural networks, could effectively be applied to incorporate logical constraints into neural network learning. More specifically, we design a systematic way to represent discrete logical constraints as a loss function; minimizing this loss using gradient descent via a straight-through-estimator updates the neural network’s weights in the direction that the binarized outputs satisfy the logical constraints. The experimental results show that by leveraging GPUs and batch training, this method scales significantly better than existing neuro-symbolic methods that require heavy symbolic computation for computing gradients. Also, we demonstrate that our method applies to different types of neural networks, such as MLP, CNN, and GNN, making them learn with no or fewer labeled data by learning directly from known constraints.} }
Endnote
%0 Conference Paper %T Injecting Logical Constraints into Neural Networks via Straight-Through Estimators %A Zhun Yang %A Joohyung Lee %A Chiyoun Park %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-yang22h %I PMLR %P 25096--25122 %U https://proceedings.mlr.press/v162/yang22h.html %V 162 %X Injecting discrete logical constraints into neural network learning is one of the main challenges in neuro-symbolic AI. We find that a straight-through-estimator, a method introduced to train binary neural networks, could effectively be applied to incorporate logical constraints into neural network learning. More specifically, we design a systematic way to represent discrete logical constraints as a loss function; minimizing this loss using gradient descent via a straight-through-estimator updates the neural network’s weights in the direction that the binarized outputs satisfy the logical constraints. The experimental results show that by leveraging GPUs and batch training, this method scales significantly better than existing neuro-symbolic methods that require heavy symbolic computation for computing gradients. Also, we demonstrate that our method applies to different types of neural networks, such as MLP, CNN, and GNN, making them learn with no or fewer labeled data by learning directly from known constraints.
APA
Yang, Z., Lee, J. & Park, C.. (2022). Injecting Logical Constraints into Neural Networks via Straight-Through Estimators. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:25096-25122 Available from https://proceedings.mlr.press/v162/yang22h.html.

Related Material