Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

Burak Bartan, Mert Pilanci
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:694-704, 2021.

Abstract

Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent literature, semidefinite lifting, and Grothendieck’s identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality provably in polynomial time in all relevant parameters via tight semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-bartan21a, title = {Training Quantized Neural Networks to Global Optimality via Semidefinite Programming}, author = {Bartan, Burak and Pilanci, Mert}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {694--704}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/bartan21a/bartan21a.pdf}, url = {https://proceedings.mlr.press/v139/bartan21a.html}, abstract = {Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent literature, semidefinite lifting, and Grothendieck’s identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality provably in polynomial time in all relevant parameters via tight semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method.} }
Endnote
%0 Conference Paper %T Training Quantized Neural Networks to Global Optimality via Semidefinite Programming %A Burak Bartan %A Mert Pilanci %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-bartan21a %I PMLR %P 694--704 %U https://proceedings.mlr.press/v139/bartan21a.html %V 139 %X Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent literature, semidefinite lifting, and Grothendieck’s identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality provably in polynomial time in all relevant parameters via tight semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method.
APA
Bartan, B. & Pilanci, M.. (2021). Training Quantized Neural Networks to Global Optimality via Semidefinite Programming. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:694-704 Available from https://proceedings.mlr.press/v139/bartan21a.html.

Related Material